spriteful
Offline
Vancouver, BC

Hi Everyone,

I'd like to share some details about a software project I've been working on- I hope it will turn into a fun and useful tool for electronic musicians and VJ's.

It is a hybrid sequencer for simultaneously creating (chip- style) music and (pixel- style) visuals. The concept is heavily influenced by old-school sprite-scaling video games. You can create sets of 2D pixel-art 'cards', then stack the cards into 3D shapes called 'dragons'. Each dragon represents a musical note, and contains a graph of time-varying music parameters (similar to the concept of an Instrument Definition in a tracker). By placing the dragons into a 3D world (in real-time or step-time), you can build up an entire tune.

As the final step, you will be able to interact with your composition by playing it like a video game-- jamming, changing notes, and applying effects to create a live performance.



Currently I am building a prototype in Processing.
Although it's in very early stages, it already seems fun and promising to play with.
Here is a screenshot, just a teaser really. It will probably seem cryptic since it's unfinished, but perhaps will give a feel for where things are going:




This will be available first as Windows/OS X compatible software; and eventually I intend to release it for iPhone/iPad/Android as well -- I think this type of software will be very well suited to tablet/touch input.




Some more details of the design:

-music engine to be similar to NES or gameboy audio capabilities (2 or more pulse channels with pwm, triangle, noise, 1 or more wavetable channels)
-graphics to be based on fairly low-rez sprites, probably 8x8 and/or 8x16
-sequencing engine based on a 60hz framerate for both audio and video
-I'm thinking about ideas for data interchange with Famitracker, LSDJ, etc.??
-midi input and gamepad input to be included



This thing is in early stages but I'm making a lot of progress on it these days, and I'm really excited to get some feedback. If the forum is interested in this type of thing, I will post further developments on this thread. At some point, I will also be looking for beta testers. I am very interested in collaboration, hacking, scripting, customizing etc., so I am trying to figure out the best model for release-- freeware, shareware, open source, or similar. (And I appreciate any suggestions regarding which approach is best.)

Until now, I haven't shared the idea with anyone but a couple of close friends/collaborators; so I'm dying to know- what do people think of this concept? Would you be interested in using this type of thing? Are there any 'must-have' features you think I should keep in mind?

All questions/comments are welcome.

herr_prof
Offline
Unsubscribe

looks really sweet.. keep us posted!

EBOT
Offline
venezuela ccs

super cool project man!, i´m working with similar software to visual experiments. i need to see your final result!!!   wink

Last edited by EBOT (Oct 6, 2010 11:35 pm)

Battle Lava
Offline

Right on man, this looks REALLY cool. I'm very interested in beta testing/collaborating/giving suggestions, if you like. Wicked idea.

jsr
Offline
cph

absolutely cool!
it's funny -- i just started working on a processing tracker too (although much more classic in its approach) -- i am writing it primarily to program instruments + small patterns for my hardware synth CHEAP, FAT and OPEN -- so i am sending all commands over serial to generate sounds from the synth. the plan is then to upload the instruments to CFO, and then jam them from that.
but hey! this is cool! perhaps we could somehow feed off each others projects? maybe your tracker could send commands to my synth?
check it: http://tthheessiiss.wordpress.com/
and a small demo of the tracker: http://www.youtube.com/watch?v=mUDA0CXZ9ME

Piksel Blue
Offline
Jessup, Maryland

cool its like seeing sound

goto80
Offline
CHIPTUNE

Nice!

As you say, would be great to be able to import songs into this (since there are still irritatingly few options for tracker music & visuals). I'm not only thinking about NES/GB-songs, but perhaps also other tracker formats?

I would encourage you to go beyond the NES/GB sound capabilities. To me it seems like a waste of potential to restrict it like that. Maybe include support for 4 channels of 8-bit samples (to play mod-files, for example), or add some rudimentary synthesizer with more than squarewaves. Maybe add some generative features for composition aswell, just to get even further away boring tracker dogmas... hehe.

Either way -- great project! (PS, collaborate with JSR because he does good stuff smile)

spriteful
Offline
Vancouver, BC

Thanks guys, for the encouragement and suggestions. smile
It's almost surreal, in a good way- In the past I've been kind of a hermit when it comes to music/art projects, but I really wanted to break that mold this time and collaborate with the community at large. And I've been nursing this concept in the dark for over a year and a half (with some important collaboration from one of my best friends who is an artist, musician and huge retro-gaming expert). Thus, receiving even a few comments like this feels like coming out of a cave into the bright light... wow!

herr_prof, EBOT, piksel blue- thanks! I'll keep you all posted.

Battle Lava- sounds good, I will remember your interest when I get to the next stage (soon I hope).

jsr- Yes, in the long term I would *love* to make my tracker control actual hardware. Computer control of hardware is a delightful thing (btw I am essentially broke due in large part to buying hardware synths!), and CFO looks like a great design; that is seriously impressive. I am reading more about your projects; I will think about the possibilities and stay in touch.
By the way, speaking of Processing, eventually I am interested in doing a custom audio engine (and I have one partly coded for iPhone), but for now my Processing prototype is using the CSoundo library to control CSound.

goto80- Yes, I fully agree about going beyond pulsewaves. The initial specs are basically a way for me to actually get something up and running; and I wanted something that would be basically a _superset_ of NES/GB channels because it will be cool to tap into peoples skills and interests in those formats, and perhaps give people a way to visualize existing tunes.
In the long run however, my ideal vision would be to give the software its own personality... not just 'boring 16-bit / 44.1kHZ samples', but not just being slavishly retro either. Something that meshes elegantly with the audio hardware it runs on and the graphical component. I've been thinking about the potential of wave-sequencing synthesis, using waveforms that are low-rez enough to be human-drawable.The idea then would be that each 'card' has bitmap graphics on the 'front' and a drawable wavetable on the back; there is definitely a pleasing harmony to that concept. In fact I was up late last night researching the audio capabilities of the TurboGrafx/PC Engine, because that is imho a funky and underrated system as far as music, and it has kind of a quirky 'mid-range' wavetable spec. ... come to think of it I was also reading about Context Free Art and Structure Synth late last night, so, I think we are 'smoking the same stuff' so to speak wink

Well, I better get back to coding, and take advantage of all the good vibes and momentum!!

tacticalbread
Offline
))<>((

Definitely looks pretty sweet.

And I love Processing. heart

wedanced
Offline
San Francisco

niiiiice

Battle Lava
Offline

You gotta start a blog or twitter account or something about the project so anyone interested can follow up on what's happening.  I really want to try this thing out!

spriteful
Offline
Vancouver, BC

Project update-- I'm making lots of progress on the SpriteMusic sequencer and I think a demo video will be coming shortly. Then after the video release and a little more feature additions/cleanup, I will be looking for test users. Most of my development time since last posting has been spent on the audio engine and audio/video sync... even though I am still working at the 'alpha/prototype' stage it is important to me to have the audio quality be good enough to make something cool with.

After testing with both CSoundo and Pure Date via OSC, I have switched over to the 'Minim beta' sound library for various reasons, which has the benefit of being fully integrated with Processing. Currently the audio engine is based on 5 bandlimited-wavetable oscillators, each switchable between pulse wave, triangle, noise, saw, sine and parabolic waveforms. Each instrument allows 60Hz modulation of (at least) Amplitude, Pitch (fine and arpeggio), Waveform (for PWM and wave-sequencing), and Pan.
Eventually there will be some form of user-drawable waves, and I may just throw in some 'regular sample playback' channels as well, for users who want to make 'chiptune plus edm drums' style tracks.

(Edit: I forgot to mention... I have been heavily tempted also to bump up the audio control-rate to 240Hz. Not *just* cause I am inspired by all the Pulsar talk {Hi, Neil Baldwin!} but also because after re-reading a bunch of NES APU docs I realize that it has more capabilities than I thought for modulating things at that rate. On the other hand there is something magical in the 30/60Hz range where both the eye and the ear can detect changes, leading to the synaesthetic feeling I strive for. Also I'm reluctant to decouple the nice 'one audio moment per card' paradigm. hmm.)


I would like to have choices for different audio engines to make this more widely useful/appealing: for instance I will probably look into using Blargg's audio libraries for accurate NES-music playback. And output to hardware synths is still on my mind as well.

Here are a couple of new screens to look at, picked from nice moments in testing. Actually I've had these for awhile, just didn't get around to posting. It does seem a little funny to keep posting screenshots when the audio and the animation is such an important part; so that's the reason I hope to be posting videos any day now. (Well, realistically, within a week is certainly possible.) And once I get out some videos I will endeavor to explain many more details of how you use the software.

1- This shows a newer version of the interface, and demonstrates multiple types of 'dragon' onscreen. It also demonstrates graphics which 'look like something', rather than being abstract. It was done by a test user (family member).

2- In the bottom panel, you can see the audio adjustment mode, where parameters are set on a 'per card' basis.

3- When saving files, the graphical part (sprite sheet) is saved to PNG file, so it is fun to look at them and potentially you could edit further with Photoshop, etc.

4- An older test shot, this is an abstract thing I made; it looked cool in motion.

The software seems to be running fine on both my Mac and Windows test machines. It's possible to slowdown the graphical framerate with too many sprites on screen, but on the other hand I am only still using OpenGL in 'direct mode', i.e. not really taking advantage of the graphics cards. Therefore I guess that lots of further optimization is possible.

One thing to clarify is that the final version will have many options for automatically linking audio parameters to graphical adjustments. For instance, pitch >> height, pan >> xpos, scale or chord >> color, perhaps volume >> opacity, etc. But a lot of that linkage (and for now, most of it) is just 'done manually', so basically you can explore how you would like to link different sounds/modulations to different images. A deep hope I have for the project is that once I release this we can all have some really interesting conversations about 'how does one visualize music'.

Last edited by spriteful (Oct 28, 2010 9:05 am)

BetaSynapse
Offline
Cambridge, MA

Why do the sprites angle upwards in the sheet? It would make more sense if they were affixed to cells. Also, could we specify tile sizes for the sprite sheet? So then you would not be restricted to retro sprites in the actual program. It would add a lot more visual functionality methinks.

You should also have a Default setting for linking image parameters with audio ones, but still allow manual control for things. Like I might want pitch to correlate to color, and volume to height. I'm just a huge fan of having a simple default interface, but then more complicated custom options if you want to muck around. Looking excellent! I'd love to test this once you release! big_smile

spriteful
Offline
Vancouver, BC

Hi BetaSynapse,
Good question about the sprite sheet. Actually the sprites don't angle upwards, and they do fit in a regular grid-- it's just a coincidence that these particular sprites were drawn in this order. For example, the 'water' sprites were created first and happen to use the lower portion of the grid; later on the clouds were drawn and happen to use the upper portion.

Something I didn't mention yet about the sprite sheets: I will be trying to add a feature to include 1 or more 'alternate sprite sheets' which would allow for instant palette swaps, or image-filtered versions of every sprite... this would make a good correspondence with applying a global audio effect.

I know what you mean about tile sizes. Yes, there will probably be some different options for different sizes. Design-wise, there is an interesting issue here which is that I want to strike the right balance between giving my software some 'personality' yet still letting users express their personal style with it. I want it to be a bit like playing a game, not just like using Photoshop or some other 'editing tool'. Actually in a perfect world I would be able to release different versions almost like 'skins', ie the '8-bit-style' version, the '16-bit-style', etc. The prototype will probably be more limited than what comes later, to let me start by 'doing one thing, but well'.

Regarding the image->audio link... yes, the approach you describe makes a lot of sense and is similar to how I've been thinking about it. In fact, there will probably be 3 modes for the lower 'editor panel': sprite drawing, audio parameters, and 'misc' including various mapping options. Eventually I would like to allow for really complex things such as modulating graphics based on harmonic relationships (e.g. Key->Chord->Scale Degree).

Also, to talk again of 'the ultimate plan', I can imagine at least 4 different levels of sound<>image linking:
1. manually drawing graphical frames which 'illustrate' the audio frame
2. default or optional auto-linking as we are discussing
3. tools for making a N-dimensional grid of 'variants' for each instrument, corresponding to different but related note inflections... and maybe use e.g. an x/y joystick to select between these
4. 'channel automation' (similar to pattern fx in a tracker)... a way to further modulate the parameters of placed notes, with further graphical/audio linking

spriteful
Offline
Vancouver, BC

That video is ~allllmost~ ready... but in the meantime, here is some actual audio output from the SpriteMusic software. It really does make sound smile !

http://soundcloud.com/spriteful/spritemusic-soundtest1

As mentioned, I will probably incorporate existing libraries later for accurate NES emulation, but this is a taste of the built-in engine. I'm starting to get happy with it as it is in some ways reminiscent of C64 sound (and maybe a bit of Nanoloop) but has its own flavor.

I have not put in any filters for now, but you can do cool things with PWM and with rapidly switching waveforms.

p.s. This is not an example of my 'composition' abilities, I hope, I was experimenting mainly with sound parameters wink .

spriteful
Offline
Vancouver, BC

Well this is not the 'big unveil' video yet but I'm excited about how SpriteMusic looks in motion so here is a bit of footage (video only for this demo, no sound). If you watch the whole video you will see two different animation modes: 'frozen motion' curves, followed by 'dancing' modulation. (These are not the only modes smile )

Say you have an instrument with some vibrato on pitch: you can turn on the 'pitch tuning --> sprite height' linking option and the vibrato will be visualized as bouncing curves.

You can also see how I'm working on a system to modulate color in several dimensions according to the music.
Further options include transparency, scaling, etc.

Here it is:
http://vimeo.com/16650892

I'm still working on the first 'official' video since I want it to be ~nice~. When you see the music and visuals working together it is all going to make sense!