HawkEye

Inspiration

Our grandparents and elders are some of our best friends, and teachers. But often times we forget, who's there for them? Not just our own, but many alike around the world don't have someone to be their caregiver. However, this problem isn't just prevalent in our senior community. Those with visual impairments or mental disabilities also lack aid when they should . That's why we created HawkEyes, an AI servicebot/caregiver that contains the emotional intelligence and logic capable of doing what animals and humans can do, and more.

What it does

HawkEye is a pair of glasses/headset that acts in place of a service animal or caregiver. It can perform mundane tasks like counting money, to some pretty crazy tasks like teaching how to use YouTube, to some even crazier tasks like having conversations about the meaning of life in German. It also makes calling (FaceTiming) and reaching other people much easier, being able to call your loved ones or emergency services with a single command. HawkEye also features two modes; these being active and passive monitoring. With its passive mode, HawkEye periodically scans and detects people in front of the user, alerting them of where obstacles might be. This is a feature dedicated to those with visual impairment, helping them navigate their day to day lives. With its active mode, HawkEye's capabilities grow tenfold, featuring Lucky, our humanized AI assistant. 

How we built it

Frame:

Using recycled materials and adhering to a simple but effective design, we created HawkEye with ergonomics and sustainability in mind. Most of the wires are hidden within a sturdy frame, covered in black card stock for a clean look. But what's life without a little fun? In the case that the user feels the need to add a little bit of flair to their pair of HawkEyes, we've made the decision to allow magnetic add-ons. These can be attached to the side of the glasses so as to not interfere with vision. All this extra weight can cause strain- so to balance each side, we distributed the weights of the speaker, microphone, and ports evenly for a comfortable wearing experience.

AI:

HawkEye is comprised of 4 different AI modules. To pick up speech, we used AssemblyAI API which allowed us to accurately perform STT for the voice commands. Then these commands are processed and sent to the brand new model GPT4o for processing images and conversations. If HawkEye is in it's passive mode, we used openCV and a CocoSSD object detection model to count and find the number of people in front of the user, alerting them every once in a while where they are going. Finally, Lucky (HawkEye's AI assistant) has a voice of gold that is only possible through OpenAI's TTS allowing conversations and requests to sound more genuine and human.

Challenges we ran into

Figuring out the implementation of hardware with the design of the glasses was the most time consuming part of the project, having to figure out how we can cram loads of tech into a wearable solution.

Accomplishments that we're proud of

Lucky (the AI assistant's name) is different from normal AI assistants that you might find in a chatbot. We developed Lucky's personality to be as human and genuine as possible, taking down the barrier that sets apart AI from humans. Its ability to also answer even the most absurd questions with a bit of joy and humour gave us a glimpse into the potential of HawkEye's implementation into the real world.

What we learned

Throughout the processes of construction and programming, we picked up new skills and technologies.
1.  AI Speech to Text

2. How to use an accelerometer

3. Thermal cameras and their safety uses 

4. Computer Vision 

What's next?

One of HawkEye's star features is its ability to be modular. Leaving the circuitry exposed with patches of magnets along the frame allows the user and communities to develop and add on different technologies based on their needs. For example, we brought 3 types of modular pieces that we saw the potential in. One of these was an accelerometer and gyroscope that could help elderly and those prone to falling automate an emergency dispatch to them no matter their state. Another piece was an ECG that could record and measure heart rate, constantly monitoring vitals. Finally, for the most ambitious piece, an EEG that reads brainwave activity to make sure people with autism always have support despite facing challenges communicating their emotions.