Audio-Visual "SpriteMusic" Sequencer: new software in development (Page 1 of 2)

looks really sweet.. keep us posted!

Right on man, this looks REALLY cool. I'm very interested in beta testing/collaborating/giving suggestions, if you like. Wicked idea.

cool its like seeing sound

Thanks guys, for the encouragement and suggestions.
It's almost surreal, in a good way- In the past I've been kind of a hermit when it comes to music/art projects, but I really wanted to break that mold this time and collaborate with the community at large. And I've been nursing this concept in the dark for over a year and a half (with some important collaboration from one of my best friends who is an artist, musician and huge retro-gaming expert). Thus, receiving even a few comments like this feels like coming out of a cave into the bright light... wow!

herr_prof, EBOT, piksel blue- thanks! I'll keep you all posted.

Battle Lava- sounds good, I will remember your interest when I get to the next stage (soon I hope).

jsr- Yes, in the long term I would *love* to make my tracker control actual hardware. Computer control of hardware is a delightful thing (btw I am essentially broke due in large part to buying hardware synths!), and CFO looks like a great design; that is seriously impressive. I am reading more about your projects; I will think about the possibilities and stay in touch.
By the way, speaking of Processing, eventually I am interested in doing a custom audio engine (and I have one partly coded for iPhone), but for now my Processing prototype is using the CSoundo library to control CSound.

goto80- Yes, I fully agree about going beyond pulsewaves. The initial specs are basically a way for me to actually get something up and running; and I wanted something that would be basically a _superset_ of NES/GB channels because it will be cool to tap into peoples skills and interests in those formats, and perhaps give people a way to visualize existing tunes.
In the long run however, my ideal vision would be to give the software its own personality... not just 'boring 16-bit / 44.1kHZ samples', but not just being slavishly retro either. Something that meshes elegantly with the audio hardware it runs on and the graphical component. I've been thinking about the potential of wave-sequencing synthesis, using waveforms that are low-rez enough to be human-drawable.The idea then would be that each 'card' has bitmap graphics on the 'front' and a drawable wavetable on the back; there is definitely a pleasing harmony to that concept. In fact I was up late last night researching the audio capabilities of the TurboGrafx/PC Engine, because that is imho a funky and underrated system as far as music, and it has kind of a quirky 'mid-range' wavetable spec. ... come to think of it I was also reading about Context Free Art and Structure Synth late last night, so, I think we are 'smoking the same stuff' so to speak

Well, I better get back to coding, and take advantage of all the good vibes and momentum!!

niiiiice

Project update-- I'm making lots of progress on the SpriteMusic sequencer and I think a demo video will be coming shortly. Then after the video release and a little more feature additions/cleanup, I will be looking for test users. Most of my development time since last posting has been spent on the audio engine and audio/video sync... even though I am still working at the 'alpha/prototype' stage it is important to me to have the audio quality be good enough to make something cool with.

After testing with both CSoundo and Pure Date via OSC, I have switched over to the 'Minim beta' sound library for various reasons, which has the benefit of being fully integrated with Processing. Currently the audio engine is based on 5 bandlimited-wavetable oscillators, each switchable between pulse wave, triangle, noise, saw, sine and parabolic waveforms. Each instrument allows 60Hz modulation of (at least) Amplitude, Pitch (fine and arpeggio), Waveform (for PWM and wave-sequencing), and Pan.
Eventually there will be some form of user-drawable waves, and I may just throw in some 'regular sample playback' channels as well, for users who want to make 'chiptune plus edm drums' style tracks.

(Edit: I forgot to mention... I have been heavily tempted also to bump up the audio control-rate to 240Hz. Not *just* cause I am inspired by all the Pulsar talk {Hi, Neil Baldwin!} but also because after re-reading a bunch of NES APU docs I realize that it has more capabilities than I thought for modulating things at that rate. On the other hand there is something magical in the 30/60Hz range where both the eye and the ear can detect changes, leading to the synaesthetic feeling I strive for. Also I'm reluctant to decouple the nice 'one audio moment per card' paradigm. hmm.)

I would like to have choices for different audio engines to make this more widely useful/appealing: for instance I will probably look into using Blargg's audio libraries for accurate NES-music playback. And output to hardware synths is still on my mind as well.

Here are a couple of new screens to look at, picked from nice moments in testing. Actually I've had these for awhile, just didn't get around to posting. It does seem a little funny to keep posting screenshots when the audio and the animation is such an important part; so that's the reason I hope to be posting videos any day now. (Well, realistically, within a week is certainly possible.) And once I get out some videos I will endeavor to explain many more details of how you use the software.

1- This shows a newer version of the interface, and demonstrates multiple types of 'dragon' onscreen. It also demonstrates graphics which 'look like something', rather than being abstract. It was done by a test user (family member).

2- In the bottom panel, you can see the audio adjustment mode, where parameters are set on a 'per card' basis.

3- When saving files, the graphical part (sprite sheet) is saved to PNG file, so it is fun to look at them and potentially you could edit further with Photoshop, etc.

4- An older test shot, this is an abstract thing I made; it looked cool in motion.

The software seems to be running fine on both my Mac and Windows test machines. It's possible to slowdown the graphical framerate with too many sprites on screen, but on the other hand I am only still using OpenGL in 'direct mode', i.e. not really taking advantage of the graphics cards. Therefore I guess that lots of further optimization is possible.

One thing to clarify is that the final version will have many options for automatically linking audio parameters to graphical adjustments. For instance, pitch >> height, pan >> xpos, scale or chord >> color, perhaps volume >> opacity, etc. But a lot of that linkage (and for now, most of it) is just 'done manually', so basically you can explore how you would like to link different sounds/modulations to different images. A deep hope I have for the project is that once I release this we can all have some really interesting conversations about 'how does one visualize music'.

Last edited by spriteful (Oct 28, 2010 9:05 am)

Hi BetaSynapse,
Good question about the sprite sheet. Actually the sprites don't angle upwards, and they do fit in a regular grid-- it's just a coincidence that these particular sprites were drawn in this order. For example, the 'water' sprites were created first and happen to use the lower portion of the grid; later on the clouds were drawn and happen to use the upper portion.

Something I didn't mention yet about the sprite sheets: I will be trying to add a feature to include 1 or more 'alternate sprite sheets' which would allow for instant palette swaps, or image-filtered versions of every sprite... this would make a good correspondence with applying a global audio effect.

I know what you mean about tile sizes. Yes, there will probably be some different options for different sizes. Design-wise, there is an interesting issue here which is that I want to strike the right balance between giving my software some 'personality' yet still letting users express their personal style with it. I want it to be a bit like playing a game, not just like using Photoshop or some other 'editing tool'. Actually in a perfect world I would be able to release different versions almost like 'skins', ie the '8-bit-style' version, the '16-bit-style', etc. The prototype will probably be more limited than what comes later, to let me start by 'doing one thing, but well'.

Regarding the image->audio link... yes, the approach you describe makes a lot of sense and is similar to how I've been thinking about it. In fact, there will probably be 3 modes for the lower 'editor panel': sprite drawing, audio parameters, and 'misc' including various mapping options. Eventually I would like to allow for really complex things such as modulating graphics based on harmonic relationships (e.g. Key->Chord->Scale Degree).

Also, to talk again of 'the ultimate plan', I can imagine at least 4 different levels of sound<>image linking:
1. manually drawing graphical frames which 'illustrate' the audio frame
2. default or optional auto-linking as we are discussing
3. tools for making a N-dimensional grid of 'variants' for each instrument, corresponding to different but related note inflections... and maybe use e.g. an x/y joystick to select between these
4. 'channel automation' (similar to pattern fx in a tracker)... a way to further modulate the parameters of placed notes, with further graphical/audio linking