Music and Speech

I've often wondered why we like music. It seems sort of arbitrary, and it may actually be soon (more on that later). But regardless, we do like music. It's found in every culture around the world. It is truly ubiquitous as a human trait (barring a small portion of the population whose brains are wired differently). So why do we like music?

Well it could just be a fluke. Maybe we just like music because it's something our brains just evolved to like. However I'd like to posit a different hypothesis. Of course this is just my opinion, and it could be wrong, but I think it hangs together pretty well.

Start hundreds of thousands of years ago. A branch of proto-humans starts experimenting with sound. This isn't anything new or unique. Dolphins and various birds make a whole host of complex sounds to communicate. Many animals do complex mating dances and calls. Of course many animals don't do much of this. Slugs don't do a whole lot of anything. Cats have a few different meows and hisses at their disposal, but there's only so much you can communicate with that.

So this branch of proto-humans finds that having a wider range of noises they can both make and distinguish gives them a unique ability to coordinate with other member of their species that other animals can't. As so it was that speech was selected for. We probably made different pitches of grunts, whistles, hums, and so forth. Over time this became so useful that we started developing physical structures to aid in the creation of these unique sounds.

At a certain point, this evolution of noises hit a tipping point where the ambiguity of what we were conveying was replaced with codified languages. Why have to distinguish between a high pitched noise and a low pitched noise when we can agree on logical operators which can literally convey anything (math, science, religion, emotion, etc.).

So what does it have to do with music? I like to think that music was a skill which allowed our proto-human ancestors to get better at both the making and distinguishing of sounds. I can almost imagine groups of proto-humans dancing around in circles stamping their feet in rhythum and making coordinated sounds not all that dissimilar from music. The species which practice music and had an appreciation for it were able to better recognize subtle differences and relationships between sounds that non-musical minds could not. There may well be different paths to speech that don't involve music, but I think music was a very obvious and simple tool to enable the evolution of speech.

And this never went away. But will it? We have a robust set of communication techniques available to us. Different spoken languages, art (which may have similar roots as it relates to other centers of the brain), math and logic, sign language, facial expression and so on. As far as developing a patter of making noises to communicate with other members of our species, we're pretty much “there”... right Well there's two answers I think are plausible (actually more than that, but these are the broad categories). An optimistic view and a pessimistic one.

First the pessimistic. We really have gotten about as much out of the tactic of making music as we can get. Music is nice, but it's vestigial and has served it's purpose. Over time, we may lose the ability to care about music. As we use complex languages born out of a musical background, the need for that skill will just fade away. Perhaps not completely, but it will no longer be a powerful driving evolutionary force as it was in the past leaving it to be overwritten with more applicable software.

Now the optimistic. We have very little insight into what traits actually drive our future evolution. It's reasonably easy to look at our history and see why we evolved one trait over another. But predicting where we will evolve next? There are just too many factors at play to know which one will dominate others, especially with something as complicated as the human physical and emotional reaction to beauty. Doesn't that sound optimistic? Our profound ignorance in the forces at play guiding our evolution? Well the optimistic part is that we've had complex language for a long time, and we still have music as well. It should be obvious to anyone who loves music that the parts of the psyche that are touched when listening to or creating music are deep and meaningful. Regardless of what it may have started out as, we have incorporated it into how we as humans think. It's quite possible it serves a different purpose now. Maybe more along the lines of why we dream. Or how we connect emotionally with others. Maybe it's still fine-tuning how we connect with people. It's a form of communication; of poetry. When we hear a touching lyric in a song, we contemplate it's meaning. We act on those emotions. Music may be more central to how we think than we even realize.

El Trumpo

Of all the traits of the so-called president trump, I'd like to focus on some of his personality traits which most would agree he exudes. I will then show how those traits combined with his position of power leads to a situation which causes myself and many others to be very concerned.

Donald Trump is a narcissist. He think he's the greatest thing since sliced bread. He puts his name on things in great big block gold letters. He tells us he's the only one who can defeat ISIS. He tells us he's going to bring America (back) into a golden age of power and prosperity which none before him have been able to achieve. Even assuming for the moment that all that were true, how frequently he feels the need to tell us about it goes a little bit beyond that of the average Joe.

He demands attention. Even in his previous life, The Apprentice, his twitter feed, his books, lying about being his own publicist, goes to great length to be not only noticed, but paid attention to. Positively or negatively, he doesn't seem to care.

He will disparage anyone or anything which he feels is not inline with his desires or beliefs. This is evident by the numerous lines in his tweets and speeches to the effect of “...The so-called Judge...” or “, which/who is highly overrated” (substitute Hamilton, Bill O'Reilly, The Atlanta Falcons, had they won the superbowl, Mother Theresa) is so pervasive in his speech, as a tactic he uses it about as much as he takes breath.

He does not have a problem lying, and worse has seemed to successfully wiggle out of his lies with an interpretation of reality which denies the existence of objective truth. The number of times he's said he never said something only to state, bald-faced and un-repentant that he never said it is shocking really.

He is a conspiracy theorist, and is generally unwilling to accept information which goes against his beliefs. Like all conspiracy theorists, any attempt to show him otherwise is either futher evidence of “the coverup” or “the dishonest media” or elicits a further refinement of his belief which encompasses the presented information. Take for example the “birther” conspiracy. Or the investigation into Clinton's emails. Or voter fraud. Or one of any number of similar thoroughly investigated and documented cases. The level of evidence he demands to be “proven wrong” is so far beyond reasonable that it is practically impossible (thus giving him an out to say “you can't prove that). In the birther conspiracy, he, after seeing his long form birth certificate and numerous other pieces of information demanded something along the lines of “a high-school year book from his graduating class, signed by a teacher”.

So in summary, he is a narcissistic, loud-mouthed, unhinged (to whatever degree you wish to believe) conspiracy theorist who the only thing he loves more than attention is an audience.

I tried to think of a crazier person to try to, by analogy, illustrate just how bizzare a character he is, but everyone I could think of prior to his Presidency pales in comparison. Kayne West was the first person who came to mind, but I feel that does him a great disservice being put in the same arena as Donald Trump. Maybe Charlie Sheen during his “wild” phase. Or anyone from the cast of The Jersey Shore. All these people love attention. All these people have views on the world which are on the best of days, fringe, and on the worst of days, schizophrenic. They have their spheres of influence, but it's limited.

The leadup to his election was troubling for so many reasons, but the thing that stuck with me was we gave a crazy person a megaphone and an enormous captive audience. In such a situation, how could a person like Trump not spiral out to bigger, stranger, more provocative things?

Despite that, roughly half of the fraction of people who voted decided there was something there worth exploring. To the half who didn't vote for him, we were able to see this pattern evolve and where it would go. Now, not only did we give him a megaphone and an audience, we gave him the loudest megaphone and the largest audience possible (the world), but we gave him power. Offhand remarks must now be actually considered for their implications because the person making them controls the largest military in the world.

Some have said that the system of checks and balances would prevent a truly tyrannical presidency from evolving, but again, that seemed unlikely before the election and even less so now. Refer back to the start of my writing. If he disagrees with something, he will disparage, insult and circumvent until what he wants is realized. If that means signing an executive order, that's what he will do. If that means blasting anyone in government with the balls to stand up to him in every theater possible he will do that. If it means firing qualified individuals to replace them with personal friends to whom he has promised power, that's what he will do. As a single man, the things he can do with his own two arms is limited. But that is not what a President is. A president is a coordinator, with a hand in everything this government does. He may not be able to unilaterally decide to drop a nuclear bomb on whatever country he wishes, or repeal the civil rights act, but he can absolutely fire anyone in his cabinet who doesn't step in line, and make life hell for those he can't directly fire. He has already proven that. He fired the woman from the justice department for standing up to his ill-concieved executive order. He has slandered and insulted members of the public, the media, and the government. He has made life as difficult as he possibly can for anyone who opposes him and he is well within his rights to do so. Those individuals, many of which are in Congress, the house of Representatives or the supreme court are faced with the tough decision of potentially having their entire life turned upside down and dragged through the mud for the next 4 years, or shape up and kiss the ring.

What would you do? I can't say I would do any different than many are. Getting your life turned upside down as a Congressman could end your career in politics. It could ruin your marriage, children's upbrinings, financial and psychological well being. And there's always the belief which, as I said I personally feel is misguided, that things won't be that bad. This is politics. Politics is and ugly duplicitous game. Sacrifices are always made even by the best among us, to say nothing of those in the middle of the bell curve. As Pablo Escobar used to say, “Plomo o Plato”; Lead or Silver. A bullet in the head or a bribe in your pocket. Standing up to his insanity will get you a bullet to the head (metaphorically speaking... I hope). Whereas support does not go unnoticed. Trump has certainly kept many of his promises to those who helped him become elected (think Steve Bannon, Vladimir Putin). And even the lesser among those getting in line are at the very worst, be left alone. A shimmering prize for simpily turning a blind eye to the overt insanity the man preaches on a daily basis.

This is what it boils down to. We have set all the pieces in place for a very tyrannical looking 4 years. Whether we have death squads, curfews, military law, remains to be seen; I don't think we will. But as I've hopefully made clear, we have installed a government with all the pieces in place to commit serious abuses of power, and there is almost nothing we can do to stop it should we desire. We are essentially at the good will of the administration, and given who that administration consists of, we should all be very concerned.

Words to Live By

I'm gonna get all philosophical on you all for a minute and talk about the two snippets of advice/information that I think have changed my outlook on the world more than any others.

The first is a quote misattributed to many and evolved over the years to the point I won't even claim I know who said it first (however I found a very interesting article on the subject you can read on your own). With my own personal flourish of interpretation is is basically this:

Small minds talk about other people. Average minds talk about things. Great minds talk about ideas.

First before I invite too much criticism, I admit off the bat, this, at face value could be very misleading (see the article I linked above for some of those nuances). But the point I think is very true is the first of the three points: small minds talk about other people. Talking about other people is easy. Especially so when it's negative or judgmental. Take for instance, someone who's brother has a substance abuse problem. Talking with your friends about what your views on drug addicts are, or how he made a choice, or how he's thrown away his life are comments as easy to speak as breathing. Or take politics. The 2016 presidential election was a shitstorm of people talking about other people, and in truly vitriolic and detestable ways. Or my favorite (or should I say my least favorite) example; reality TV. I'm talking here about shows like "The Real Housewives of " or really any combination of the elements "spoiled rich people get drunk, yell at eachother and fuck a lot". Those shows take the small-mindedness to another level. They take all that easy-to-spew negativity and turn up the dials on it. Making it more vile, more vapid, and sadly, more entertaining. It glorifies being small minded. It encourages you to sit and bask in the shit they're spewing and feel good about it.

The second two points, average and great minds, I think part of me believes too, but with trepidation. A quote as elegant as this was written by someone clever, and so the dice are loaded towards that end of the spectrum right off the bat. But regardless, I think moving away from just bitching about other people is the right direction to go. In my own life, I hate talking about people. I admit, I'm a bit of a loner. I like computers and solo motorcycle rides across the plains. So the need to blather on about the shortcomings (or even accomplishments) of other people has never come as naturally to me as I think it does to others. But I'm very conscious when I speak about what I'm talking about. I'm taking no position here, but for example: am I talking about Hillary Clinton being a bad person? Am I talking about the security at Benghazi? Am I talking about how peace can be achieved in the middle east? There obviously has to be an intermingling of these three categories. But I feel if you have to invoke talking about a person and their character traits, it should only be along the way to a broader point which can actually affect something.

The second bit of advice is not really a quote, and I don't remember who said it to me; I think it was one of my college professors, but they told me to start counting every time you say my, me, or I in a sentence. I (1) do it a lot. We all do. When I (2) was growing up I (3) had an older brother. My (4) best friend was in his grade so I (5) spent a lot of time hanging around adults before my (6) time. Adults would always ask me (7) how I (...) was doing. What was going on in my life. How growing up was going. It became clear to me as I grew up, that my natural tendency was to talk about myself. But it struck me that the adults in my life were not. They were asking me how I was doing. They were not talking to me about their days. What their jobs were like. What was on their minds. This always struck me as odd. When I was told to start counting personal pronouns, a light went off in my head. Learning to open up and care what other people are talking about is invaluable. I have many friends who after not seeing them for years will spend an entire evening talking about themselves, never once asking how I'm doing. This doesn't insult me; I understand the tendency. It just makes me a little sad that they're not aware enough to realize that they're talking to another person with the same hopes, fears, desires and stories as they have, and all it would take is a simple "and how are you doing?" to unlock those, and solidify the friendship.


What does it all mean? Don't talk about other people, unless you absolutely have to. If you must, try to think about the circumstances of those people, and not just the person as an isolated system. And rather than talking about others, if you find yourself talking to others, try to avoid saying I. You'll be shocked how much you do it, and how much you'll learn from and about the person to whom you are speaking.

Changing Shift Lever on 2013 Ninja 300 (EX300) (With Pictures)

Replace Ninja 300 Shift Lever

One of the architectural flaws of the 2013 ninja 300 (I haven't ridden later models, but maybe it's still the same) is that the shift lever seems to protrude out from the bike in such a way that if it even tips over, it can buckle and destroy your shift lever. I've had to replace the shift lever twice. The first time I lowsided at around 40 and the lever twisted up on itself like a pig tail (I don't have a picture of that any more unfortunately). The second time, I set the kickstand down and it happened to land right on a random bolt on the ground causing the bike to tip and fall over on the left side. Everything else was fine, but again, the shift lever got bent and unusable

It has been suggested to me to try to bend it back into place, but there are a few problems with that. First, any time you bend metal, it stresses it, making it weaker than it was before. I don't know by how much, but that's a consideration. Second, I tried to, and I couldn't get the thing to budge. I even tried heating the metal a bit but it still wouldn't go. If I had more tools and experience maybe that would be a viable option. But a new shift lever is about 40 bucks, and the tools to properly fix it are probably much more.

This is actually a pretty easy procedure, but if you've never done anything to a motorcycle before (which I hadn't the first time), it can still prove daunting to start taking pieces off your baby.

What you need:

• 1 OEM Ninja 300 Shift Lever
• This is again, assuming parts as of 2013. I don't think much has changed with this assembly in recent years, but I have to give that caveat.
• The official part number is 13242-0088. 
• You can also use the shift lever off a ninja 250. I did my first replacement with a 250 lever and it worked perfectly for 3 years (until I dropped the bike on it)
• Loctite (Blue 242)
• Breaker bar and/or torque wrench
• Note: using a torque wrench as a breaker bar is not a good idea since it's a sensitive tool. That said, with the factory Loctite on the bolt, you'll have to give it quite a bit of force to take it off.
• I just best-guessed the tightness of the bolt, but the official service manual says the torque of the bolt should be 12Nm (106 in-lb).
• Allen wrench adapter for breaker bar/torque wrench.
• Sorry, I don't know what size the bit is. Most Allen wrench bits come in a set. 
• Adjustable Wrench
• Again, sorry I don't know the specific size of the bolt. YOu don't need anything crazy here though. The nut it takes off is easy to move
• Rear Stand (OPTIONAL)
• I have a set of spools and a rear stand I use any time I'm working on my bike. It just gets it up off the ground a bit, stabilizes it, and doesn't lean towards me. However this is not necessary to do this procedure. Just a nicety.

Getting Started

I think I may have gone into more detail than is necessary for this, but my goal was to eliminate just about any ambiguity there could bein the procedure. The overview of the process is:

1. Remove shift lever bolt
2. Loosen nut on connector rod, and by hand, unscrew shift lever from rod
3. Screw the new shift lever onto the rod, and tighten the nut
4. Thread the shift lever bolt through the head of the new shift lever
5. Add a little Loctite to the bolt, make sure the shift lever is oriented correctly, and screw the lever and bolt back on to the bike (tighten to 12Nm of torque if you have a torque wrench)

Detailed Instructions

1. With your Allen bit adapter in your socket wrench, start undoing the shift lever bolt. You'll need to give it some good torque to break the Loctite, but once it gets going it should be pretty easy. As it comes out, you'll see washers on both sides. Don't lose these

   

   You'll need to give this some muscle

2. When you've entirely undone the shift lever bolt, you'll be left with your broken lever dangling from a little bar. This bar directly connects to the mechanism which makes your bike shift gears. It's an interesting little bit of engineering which you can play around with if you want. However, as I discovered on my first shift lever change, you don't need to touch anything but the one nut on the side closest to where the rod connects to the shift lever.

   

   Shift lever right after it comes
   off the bolt assembly

   
   

   

   Just hanging out...



Here's the bike-side of the bolt assembly.
I just blew some compressed air
in there and a quick rub from a rag to
clean it up

3. With the shift lever bolt detached from the bike, you now need to remove the broken shift lever from the rod. Locate the nut on the rod right up next to the shift lever. The goal here is not to remove it, but to loosen it a bit so its easier to unscrew the shift lever from the rod.

   

   In this picture, I've already loosened
   the nut on the rod a bit, so it's
   not pushing against the shift lever.

4. Now just use your hands and unscrew the shift lever from the rod. Here's a bunch of pictures of the lever as it's removed, and the bolt disassembled.
   
   
   
   

   

   Right after removal from the rod

   
   

   

   New lever (left) compared to bent lever (right)

   
   

   

   Bent lever with the bolt still in

   
   

   

   Bent lever with the bolt removed

   
   

   

   New lever with the bolt I removed
   from the bent lever in it.

   
   

   

   Another view of the new lever
   with the bolt in

5. Now we basically do everything in reverse. Use your hand and screw the new lever on to the end of the rod where the old lever was just taken off. 

   

   Lever attached to rod before
   tightening nut

   
   

   

   Lever attached to run after tightening

6. Thread the bolt through the head of the new shift lever, making sure to include the washers on both sides (you didn't lose them, did you?). Add a little bit of blue Loctite to the business end of the shift lever bolt. Orient the shift lever correctly, and loosely screw the bolt back on to the motorcycle. This can be a little confusing because the combination of the rod, the joints, and the ways you can orient everything in 3 dimensions means you can move this thing all around, and you might have even forgotten to which way to orient things to get it back on the bike. However I show the correct orientation below so if you get stuck, you should be able to emulate that. With the new lever on the rod, tighten the nut down  again. I ended up taking the bolt out temporarily while I reoriented the lever so don't be confused by the pictures with it missing. You can thread the bolt through the head it at just about any time up till you put Loctite on it.

7. 

   Threading the bolt back through
   the head of the shift lever (see
   the washer on the ground?)

   

   This is BACKWARDS! If you end up
   like this, don't worry. Just fiddle with
   all the ways you can orient the lever
   and rod until it's correct. In other words,
   so the foot-pad of the lever is pointing out 

   

   This confusing angle just shows
   how many directions this assembly
   can move it. This is me moving it
   from what you saw above (with
   the foot-pad pointing in) to the
   correct orientation (pointing out)

   

   Correct orientation 

   

   Add Loctite. A little goes a long way

   
   

   

    
   
   
8. Finally, if you have a torque wrench, tighten the bolt to the specified torque (12Nm). Otherwise, I just sort of felt it out. Tight but not too tight, ya know? 

   

   Showing how the shift lever should
   be oriented when you start tightening it

   

   Use a torque wrench if you have it

   
   

   

   Finished product!

Hope you enjoyed this!

Wildstar

Poor Wildstar. A delicious MMO undercut by some bad luck and doomed to be released in the same century as World of Warcraft. I recently started playing it again, hopeful that somehow me, little old me, by sheer willpower alone could help revitalize the community. I started leveling a character again, and in light of never being able to get into a dungeon queue, I logged on to my Warrior (or whatever the hell the big sword wielding smashy class is) who, before quitting years ago, had a burgeoning tank set. I queued for an instance and it popped. The other 4 members of the group started running off somewhere they clearly already knew, and I followed suit, while adding "Hey guys, I haven't played in quite some time, so let me know what I need to be doing). At the time, and don't ask me how, I didn't have a pair of gloves, but other than that, had mostly epics, and gear specific to tanking. But despite me assuring them that I wasn't a noob, and that I'd do just fine without a pair of gloves, they did nothing but bitch and moan about my gear, and refused to just point me in the right direction and help. And now, I know the game is doomed. It would be one thing if it were going through a rough patch, but this sentiment was not unique among this group. If you have tons of people queueing for instances, you can become accustomed to having things be just so. But in a game where there are already few players queueing for random groups, it does no good to push players away. It won't magically cause more skilled players to just show up out of the blue, and it won't encourage players to gear up, learn instances, and want to engage THEIR friends in the game. This has nothing to do with the company that makes Wildstar, but more a self defeating attitude of its players. Refusing to change their attitudes and work to better the community rather than simply complaining as the titanic goes down.

Kaleidoscopics

I got to thinking about brains today, which as any person with an overactive imagination knows, is sort of a black hole. But regardless, my thoughts shifted to consciousness, understanding of the world around us, and psychedelics (specifically DMT and other chemicals which occur naturally in the brain).

I'm not one to believe that much (if any) real astounding wisdom can be derived from the use of drugs, but it does raise the question of why such chemicals naturally exist in our brains, and why adding more of them (or sympathetic chemicals) has such a profound effect on us, perceptually, spiritually and emotionally.

As I started to think about the problem from a programming perspective, I wondered what their role could be and I came up with the idea of "kaleidoscopics". Really this is just a made up word that I though sounded cool, but the idea is that it's an agent which casts doubt upon normal inputs.

 It's difficult  to design a system which is both deterministic and "intelligent" in the sense which humans typically ascribe to the word. A simple system would be an if-then statement. More complex systems introduce more and more variables, until you have a system, which, like a brain, or an ant colony, takes on emergent properties. There's obviously a lot more going on than the simple interaction of one method or another, but to some extent after a time a system (we'll use the brain as the example from here on out) reaches a state of relative homeostasis.

So I'm rambling a little bit. The point is, how do you program something which can learn any language, lie, love, play, laugh, calculate, trust and kill? You've got to have some pretty flexible rules around what defines reality.

So my hypothesis (like most of the hypotheses I make, being grounded in absolutely no scientific study) is that these chemicals, or if not them, some other set of chemicals in the brain, has the job of turning things on their heads; randomly flipping bits, and presenting the mind with alternative interpretations of what's going on. As a child, when the brain is young and the neurochemical soup of  grey matter starts to solidify into a brain and consciousness, benefits greatly from the ability to try out theories and discard them at a moments notice.

As we get older, we start to know what the flipping of a bit will do (what if I COULD put my hand through the wall), and some of our experimentation, sadly, ends.

It also brings up the somewhat terrifying concept of The Singularity, in that to program a mind with the same ability to perceive reality as flexibly as we do, we would have to take the safety off, and really give the program the ability to come up with whatever it wants. It's pretty cool when you see it on the scale of Conway's Game Of Life, but scale it up to (super)human intelligence and allow the possibility for it to veer off in a direction like Skynet, and the human race would be in trouble. Or we could end up with something closer to Peter F. Hamilton's concept of the singularity; a more benign entity whose motivations are beyond our comprehension.

Again, I'm rambling. But if I can try to tie a nice bow on this, is that reality is highly subjective. If you think about how YOU would try to program something to "perceive" reality, it cascades into all sorts of deliciously complex issues. But one of them I'm sure is a mechanism for self doubt. It plagues us and drives us at the same time, but always has a profound effect.

Normalization Musings

Musings on Database Normalization

Much of the work I do has to do with formatting the presentation layer of data. The typical data flow is (a) vendor sends data, (b) we transform and store it in a format somewhere between what they send us as what is required by the display later (website) and (c) procedures are written against this data.

Having the luxury of placing the burden of data integrity on the vendor, in combination with an asynchronous replication model means that most of the time we cannot and do not use surrogate keys or foreign keys. Some of the time this works fine; actually most of the time it works fine. Of course if the data consumption architecture deviates from the simple truncate/insert correcting data can be vexing.

I had my first taste of self-inflicted bad database design on a recent project. Hindsight being what it is, I would have done many things differently (not the least of which would have been more planning before developing), but as it is, it was a good learning experience.

I was given a data set which I didn't understand well and a limited amount of time to develop against it. The data sets were effectively fact tables from a data warehouse; some with provided dimension files and some without. (By the way, for reasons I wont get into SSAS was not an option to use). The data sets were also quite large (fact tables around 100 million records with all the different dimensions factored in). To add insult to injury, the default format the data came in was name-value pairs rather than columns. (i.e. {Name: Price, Value: 3.14} as opposed to Price: 3.14}.

Without using surrogate keys, a key might easily have 8 columns in the primary key including a generic “Name” column catapulting the number of records by a factor of however many members there were in the table.

There were quite a few revisions I undertook to optimize both retrieval and storage. The first was to do away with the NVP structure and use dedicated columns. Sure, new values required a little bit of dev work to hook up, but even if it were a schema-less NVP format, some amount of work would still need to be done to hook up the new data points. So that cut table sizes dramatically. In combination with data compression, the data set was not looking quite manageable.

But that’s not the main point I want to make (although it is a good example of why NVP models suck).

The next step I took (and what I should have done from the beginning) was to really get to know the data, understand the natural hierarchies and unique entities. Since not all of this data was provided as reference data by the vendor, I built a process to build inferred dimension from the incoming files and maintain surrogate identifiers as necessary.

What I ended up was two conceptual layers in the database. A decomposed and normalized version of the fact tables the vendor sent, and a set of custom pre-aggregated tables to serve specific use surface data quickly (where indexing alone was insufficient). The main advantage of the normalized structure was the amount of work it removed from building and maintaining these “second-level” aggregate data structures. Being able to reference master records of data ahead of time, and logically know where to look for everything made the process of pre-populating tables significantly easier. Plus it put into my hands much more control over the integrity of the data.

The next project I worked on was a departure from the model I described in the first paragraph (vendor -> store -> present). Instead I built a database drawing upon both internal and external data sets. I used many lessons from the project I described earlier. First and foremost, I took (had?) the time to plan out the database ahead of time. Wherever possible, I normalized things out; in a few cases even so far as to have references tables with one or two id/value rows in them. I created surrogate identifiers when the need to standardize disparate data sources was needed, I documented the database, and made use of foreign keys for reasonably static reference data.

First the bad. Although to an extent, surrogate keys are supposed to be business-process agnostic, there was a need to be able to uniquely identify entities which had no natural key. As such, when setting up new items, while the value of the surrogate key was irrelevant, those identifiers made their way into descriptive models and processes. There was distinct logic written around EntityID:1 vs EntityID:2. This means that moving data from environment to environment, updating values and correcting data, these identifiers have to be maintained. MyFirstEntity must always be ID:1, and MySecondEntity must always be ID:2, not just randomly assigned numbers. Perhaps there’s a better way to uniquely identify elements with no natural key, but short of fuzzy logic, I’m not aware of one.

Additionally, foreign keys, while central to database integrity, tie your hands a bit in terms of how you can modify an immature database. For instance, during early development cycles, I was changing table structures every few day or so. Foreign Keys did their job admirably and prevented me from taking shortcuts to restructure the database and data; objectively a good thing, but it takes some getting used to.

Now the good. As with the first project I mentioned, I had to build several consolidated tables in order for front-end queries to perform optimally, but having everything structured logically made building the new structures embarrassingly simple once the foundation was configured. Highly (maybe 3NF) normalized databases requiring a wide variety of complex query format does present probme problems for systems with low latency tolerance, not all of which can be ameliorated through indexing and query design alone, but again, building de-normalized structures off of the normalized data, over which I had full control, was simple compared to trying to maintain 20 de-normalized tables and having to make sure the name of an entity never got out of sync between them.

I've worked with several highly normalized systems in the past, and while the learning curve to understand the data can be steeper than if you were looking at a de-normalized table, report or spreadsheet, once you understand the data, it makes getting what you need a much more sane process.

Foreign keys were the unsung heroes of this database. They’re not as obviously helpful as normalization or other structural advantages, but they prevent things from going wrong. Every day that I don’t have orphaned records, they’re doing their job. Having the assurance that the schema I've spent so long tinkering with and refining will work correctly is well worth the effort. Sure, it takes time to ensure I delete from tables in the correct order and don’t add sub-entities before their parent, but I doubt anyone will say the cost of learning to do things in the right order is worth having an unstable database.

There were certain instances where I made weighed decisions to avoid foreign keys (such as on tables with very heavy traffic and/or when an orphaned record would have no impact on the system, and I stand by those decisions.

I’m guessing at this point (if not earlier) normalization fans have been thinking “we told you so” since the start of the article. These are things I had no idea about when I started working with SQL. I was educated in a certain way and the methods used by my mentors just became engrained. I really had no significant exposure to highly normalized databases or foreign keys when I started, and nobody took the time to explain their implications (positive and negative), and I wish someone had. There are still many cases where I forgo these database practices and probably will again in the future, and maybe some of the time they’ll come back to haunt me, but if I can give anyone who doesn’t use these tools some advice, it would be to strongly consider them. These were tools I dismissed because others dismissed them, and I assumed the problems that arose from them were just the way things were. I found excuses not to break form, but fortunately over time my curiosity won out, and I’m glad it did. As Robert Frost said “… And I, I took the [road] lets traveled by, and that has made all the difference”.