At the beginning of August 2019, I transitioned from my previous contract placement to another with You.i TV. The focus of this project was updating the UI and UX of several established set-top box brands to match that of modern implementations, such as Netflix, Disney+, and Amazon Prime. These set-top boxes ran on different custom distributions of Linux with vastly different hardware specifications, and required writing flexible, portable code that would operate across a multitude of feature sets. During the contract, I implemented features such as:

• email integration
• main landing screen, and dynamic configuration processing
• VOD playback (for certain boxes)
• SD and HD subtitles using EIA-608, EIA-708 and DVB captioning standards
• server-sent event (SSE) management and dispatch

and worked on a variety of bugs relating to managing recordings, enabling playback, and minor device optimization for lower-end boxes. I worked with the technical lead on several occasions to help diagnose engine-level issues that prevented the application from running on the lower-end hardware, and communicated on-occasion with the engine team to provide fixes. Also, I created several testing tools using ExpressJS to mock SSE for various pieces of functionality ahead of delivery from their client and co-developers.

During the latter part of 2019, SimWave wanted me to be back in the office one day/week so I began working through the logistics of how to make that a reality. The set-top box signal comes from a coax feed hardwired through the You.i office, so getting the feed out of the office in some capacity other than video capture was not really an option. In November, during the You.i Hack Day, I worked with another coworker on a remote solution to facilitate communication with the boxes via a virtual remote, and streaming their video feed using WebRTC. The virtual remote was hosted using ExpressJS and featured a photo taken of a real STB remote with virtual button codes sent from keypresses to a backend server. The server would would send those keycodes over a serial connection to an Arduino board, which were then converted into their NEC IR protocol output and blasted at the STB. We had captured the real IR output from a remote and hardcoded the resulting codes into the Arduino to facilitate mocking the remote’s functionality.

The project was about 70% complete by the time the November Hack Day finished, and would then be revisited and completed during the following Hack Day in March of 2020. The day afterward, the announcement of the first confirmed case of COVID-19 was reported in Kanata, which prompted the company to shift to work from home. Over the following weeks, I worked with the technical and team leads for our project to scale this solution up to the entire team, which included Elgato HDMI capture cards, HDMI multiviewers, and several laptops for redundancy. I trained several members of the team with how to maintain this solution, which was still in place as I transitioned back to working at SimWave proper.

Role: Developer
Time on Project: Aug 2019 – Dec 2020
Technology used: You.i Engine, NodeJS (ExpressJS, serialport), Arduino
Language used: C/C++, Bash, JavaScript

The 3D, hands-on Electrical Troubleshooting Simulations provide an immersive environment where your professionals will learn to diagnose and repair electrical faults in complex production machinery safely, accurately, and efficiently.

During the late fall of 2018, I was brought on to assist Simutech Multimedia and their team with adapting and reimagining their 2D eletrical troubleshooting software in 3D. They had specifically asked to use Unity as their team had some experience with the engine, so we set about discussing requirements and determining what would need to be done for a successful port.

The project changed rather significantly during my time working on it; initially envisioned as a desktop application, it grew into a web app with a connected services backend to help support the lower-end hardware their clients had. I built the primary interaction framework that allowed objects through the scene to be accessible, replaced if deemed damaged, and inspected. Along with that, I was responsible for building the application’s navigation using feedback from Simutech and their clients, and facilitating communication with Simutech’s electrical solver software to properly evaluate the state of the circuit at all times.

Later, I investigated WebGL as a potential platform, and once Simutech was satisfied with that direction, we explored how best to implement the user interface for a lot of the learning content that they had present. Their design team sought to provide the learning content as HTML/CSS, so I developed a solution that would allow the same content to be presented in-editor as well as in builds, as well as integrate successfully into several popular LMS’. During this time, the team was also exploring Kubernetes as a potential avenue for managing the deployment of the application and shortly thereafter my time on the contract was complete.

Role: Lead Developer
Time on Project: Oct 2018 – Jan 2020
Technologies used: Unity, Microsoft Azure, HTML5
Languages used: C#, JavaScript

A large multimedia experience brings visitors inside the mind and body of extreme athletes to explore their experiences and motivations.

In January 2018, SimWave was approached by Science North to produce three experiences for their Beyond Human Limits exhibit, in collaboration with the Ontario Science Centre. The Central Multimedia Experience was envisioned as a hybrid quiz game/motivational experience for visitors who had visited the other zones within the exhibit. The area was designed as a hexagon, and segmented into three opposing walls, each with a projector playing a video with either educational content or positing a hypothetical question.

During the quiz section of the experience, visitors would be a question about what would be the most beneficial approach to learning a new skill and given two scenarios. All three projected walls would play a video presenting the question, then two would show the possible scenarios to achieve the desired goal. Visitors would then have to move to to the wall with the scenario they believed to be the correct answer. After a brief time has elapsed, cameras suspended in the ceiling would determine how many individuals were by either wall and select that answer. The selected answer would play on all three walls, followed by the correct answer, and a brief segment following the life and struggles of an extreme athlete.

TouchDesigner was chosen as the development platform largely because of it’s built-in video processing capabilities, and Science North had remaining licenses following a previous joint-developed project. During development, we had initially aimed to use RGB cameras to keep costs down and because we already owned licenses to software that could perform person detection on a live video feed, and would provide information back to use via a RESTful API. We were provided all three videos in a single video file, as a single computer would be driving the experience with three projectors. I’d developed a JSON configuration file that contained all of the start/end segments for each video, the correct and incorrect answers, and the overall flow of all the application so that it could be reconfigured by the staff at their behest.

Once I arrived on-site to begin installation, I noticed that the lighting in the area was very dim and found that the software could no longer distinguish people accurately. Moreover, the s0ftware was having issues with individuals wearing shirts with faces, and with individuals with backpacks. At this point, the RGB cameras were replaced by RealSense depth cameras, and the camera handling was re-written to remove dependence on the person tracking software and instead focus on blob detection via background removal. As time on-site was very limited, background removal was done via pictures taken of the area with no participants and the live feed was subtracted at runtime. As I completed my time on-site, the client had asked that children should be valued greater than adults, so using a height map, I added a valuation curve to the height and calculated the average pixel color to determine which area would be considered for the selected answer.

After installation was complete and the client was happy, I provided developer support until the contract expired at the end of the year.

Role: Sole Developer/Support
Time on Project: Jan 2018 – Dec 2018
Technology used: TouchDesigner, Intel RealSense cameras
Language used: Python, C++