"Hands Free" offers a novel way to play music. Using Apple's Vision Hand Pose, the project captures hand gestures through the device's front camera to trigger MiDi chords, letting users 'strum' melodies with simple hand movements.
For simplicity sake, I made the project only react when your thumb and pinky are touching. You can configure this behaviour by changing the code in ProcessingHands.swift.
Earlier versions of that file have all fingers enabled if you'd like to see example implementations
I re-enabled the multi finger detection. The app should play a chord when it sees your thumb in touch with the Ring or the Index finger. I'd strongly recommend checking out Rhythm Snap. The algorithm I used there is an improvement of the work done here.
So this is basically looking for a pinch action. 🤌 🤏 👌🏾
- iOS Device: Ensure you have a physical iOS device with a camera. Simulators are not suitable since they lack access to camera hardware.
- Swift Knowledge: Familiarity with Swift and an understanding of Xcode's interface are beneficial.
-
Clone the Repository
git clone https://github.com/carlosmbe/Hands-Free.git
-
Open the Project in Xcode
- Head over to the directory where you cloned the repository.
- Double click on
Hands Free.xcodeprojorHands Free.xcworkspace(if you've integrated CocoaPods).
-
Set Up Your iOS Device for Development
- Connect your iOS device using a USB cable.
- In Xcode, select your device from the dropdown list located near the play button.
- Navigate to Xcode > Preferences > Accounts. Here, ensure your Apple ID is added and that you have the correct provisioning profiles.
-
Run the App
- Simply press the "Play" icon in Xcode's top-left corner. Xcode will compile the code and install the app on your device.
- Launch: Start the "Hands Free" app on your iOS device.
- Permissions: When prompted, allow the app to access your camera.
- Get Started: Place your hand in view of the device's front camera.
- Interact: Play around with various hand gestures. The app will recognize specific hand poses and will play corresponding MiDi chords in response.
- Visual Feedback: A visual overlay will highlight the detected hand posture on the screen, and the name of the played chord will appear in the center.
Following a recent refactor, the codebase is now more organized and comprehensible:
- MiDi.swift: Central to generating and enacting MiDi Chords.
- ProcessingHands.swift: Leverages Apple's Vision framework to discern hand positions. Depending on the recognized hand pose, it then determines which chord to play.
- ContentView.swift: This SwiftUI view manages the camera interface and renders on-screen overlays.
- CameraViewController.swift: A pivotal controller. It oversees the camera's feed, coordinates Vision's hand detection requests, and interprets the detected hand positions.
Your input is valuable! Whether it's bug reports, feature requests, or code contributions, we're eager to hear from you. If you're contributing code, please ensure it's well-commented for clarity. For any enhancements or issues, please create a related issue or submit a pull request.