Tuesday, November 11, 2014

Comparison with Old Hungarian

So far I've compared the relative frequencies of initials and finals for the three most common glyphs in Rohoncian with Latin and Old Albanian. Today I'll do Old Hungarian.

For Old Hungarian, I used the four gospels from the Hussite Bible, and I counted long and short vowels together. The top three letters break down as follows:

InitialFinalAll
e, é16.5%6.1%16.6%
a, á10.1%8.5%10.7%
t6.1%13.6%8.0%

Like Latin and Rohoncian, the most common letter in Old Hungarian is more frequent as an initial than as a final, while the third most common letter is more frequent as a final than an initial. Like Latin, these two letters are e and t, respectively.

You might wonder why, if there are around 1,000 glyphs in Rohonc, I am comparing it statistically to alphabets instead of syllabaries or ideographic systems. The reason is that the most frequent glyphs in Rohonc are roughly as frequent as letters ought to be. Among Latin syllables, for example, the most common syllable in the Vulgate version of Genesis is et, but it only accounts for 3.29% of syllables. The most frequent Rohonc glyph is C, and it accounts for 12.9% of glyphs, putting it in the same ballpark as the most frequent letters of alphabetic systems.


Monday, November 10, 2014

Comparison with Old Albanian

A couple of days ago I looked at the relative frequency of the three most common Rohonc symbols as initials and finals, and compared that to the relative frequency of the three most common Latin letters.

Today I'll do the same with Old Albanian. My sample text for Old Albanian is Gjon Buzuku's Meshari, the three most common letters of which are e, i and h:

InitialFinalAll
e16.4%23.0%19.9%
i3.2%6.7%8.7%
h2.3%24.8%8.3%

In some respects, Old Albanian fits better than Latin. Latin initial i is far more common than Rohonc I (9% > 5%), while Old Abanian i shares with Rohonc I that both are far much more frequent as finals than initials. However, Albanian e occurs more frequently as a final than as an initial.

In order for this to work, the names of two of the evangelists would have to end in h. In fact, the names of two of the evangelists do end in h in the Meshari:


Maξeh: Matthew
March: Mark

Furthermore, Luke is also written with an h: Lucha.

Saturday, November 8, 2014

Relative frequency of initials and finals

In a previous post, I argued that we could use the presence or absence of hyphens at the end of a line to generate some basic statistics about word initials and word finals. At the time I was thinking of using this information to divide the text into words, but over the last few busy months I have been thinking about another use for this data.

In most (or all?) languages, the frequency of ranking initials differs somewhat from the ranking of finals and medials. For example, in Latin, the letter t occurs nearly four times more often at the end of a word than at the beginning, whereas u occurs about 3.5 times more frequently as an initial than a final.

Rohoncian is no different from known languages in that respect. For example, the glyph D occurs 8.5 times more frequently as a final than as an initial. If Rohoncian is a known real language, then the difference between frequency ranking in initial, medial and final positions could be used to help narrow it down.

For example, using the three most common glyphs in Rohoncian, we could construct a kind of litmus test. The relative frequencies of those glyphs are:

InitialFinalAll
C12.9%4.9%10.2%
I5.0%6.4%9.7%
D1.5%12.8%7.8%

If we wanted to test the theory that Rohoncian is Latin and those three glyphs are alphabetic, then we would match them up to the most common three Latin letters:

InitialFinalAll
e15.6%11.6%12.9%
i9.0%7.8%11.0%
t4.9%19.7%8.6%

In broad terms, this correspondence seems to work out well. C shares in common with e that both are ranked first in overall frequency and somewhat more frequent as initials. Similarly, D and t share the third position and are significantly more frequent as finals than initials.

The main problem with this, as far as earlier proposals go, is that two of the evangelists have names ending in D (i.e. CO IH D and XDC D). However, this is already a problem because it seems to work best to read those names as Luke (or Mark) and Matthew, and it is not clear what those names share in common that would lead them to be written with the same final.

On the positive side, I had previously proposed reading the word K O A D CX as "nights". If this is the word noctes, then the D falls in the right place, and CX could be read es. (The glyph CX looks like C, but with a dot).

Part of me wonders what we would get if we looked at initials, medials and finals in the Voynich manuscript. But that carcass has been picked over by smarter minds than mine, and yielded almost nothing.

Thursday, November 6, 2014

Quick (Tironian) Note

I've been very busy for a few months, and some of my projects have languished, including working on the Rohonc codex. (How do you prioritize a project that may not succeed?)

However, I happened to see a page from the Old Irish Book of Leinster that got me thinking about this again. I don't have much time (it's my lunch break) but I thought I could write a quick note about it.

The thing that caught my eye was the use of Tironian notes for their phonetic values. An example is the name "Conchobar", written (among other ways) as follows:


The first glyph in this name looks like a backwards C, but it is none other than the Tironian note for con:

Except, instead of representing the Latin morpheme con, it represents only the phonetic value. The same note is used in the name Conall. The RC does not look like a text that is written fully in Tironian notation, but it would be interesting to try to transcribe a sample of the Rohonc codex as though it were a subset of Tironian notation and see what it sounds like.

If you've ever wondered what a text written completely in Tironian notation looks like, here is a piece from the psalms, given at the end of a 9th century work titled Comentarii notarum tironianarum:


Saturday, September 20, 2014

The Magic Password Box

In a previous post I mentioned that an advantage of secret languages is that encryption happens within the human mind, beyond the reach of keyloggers, malware and packet sniffers.

The security of the human mind is also acknowledged in one of the most common exhortations regarding passwords: Don't write it down! Memorize it!

But memorizing passwords becomes increasingly difficult as we need more of them, and they must be more complex, and each must be different from the others. Many people have started to keep passwords in text documents, while more security-conscious people are starting to use applications like Password Safe. But what if someone gets access to your document, or there is an unrecognized vulnerability with your password storage application?

I'm in the same boat as everyone else. Once upon a time I used to generate passwords by looking around, concatenating two unrelated nouns representing things in my environment, and changing some of the letters to numbers and punctuation marks. Eventually I wrote an application to store my passwords in an encrypted text file, and that gave me the freedom to start generating passwords randomly.

Currently I probably only remember a tenth or fewer of my passwords. If my encrypted text file were lost, my passwords would be lost along with them.

Now I am experimenting with an idea that I call the Magic Password Box. The principle is relatively simple, but the affect on the security of my passwords is profound. Here is how it works:

  • I create one long password, over 100 alphabetic characters, in the form of a nonsense limerick
  • For each environment in which I need to use a password, I create a short mnemonic, like "gmail", "amazon" or "creditcard"
  • For passwords that do not require frequent updates, I compute the password as a function of SHA256(limerick + mnemonic)
  • For passwords that require updates every 90 days, I compute the password as a function of SHA256(limerick + mnemonic + quarter + year)
I'm already using a random number generator to create passwords, so replacing that with a hash isn't a huge change for me. The big change is this: I never need to store a password again, and all of my passwords can now rely on the security of my memory. If my laptop is struck by lightning, I can still get my passwords. (Perhaps I need a backup in case my brain fails to reproduce the limerick, though!)

There are some mundane considerations around how to write the code for the password calculator so that (for example) it won't leak my root password, and it can generate passwords that conform to different password policies. But there are also some interesting possibilities, such as having the calculator send the password to the system clipboard so I never even see it or type it, hiding it from prying eyes and keyloggers.

Thursday, September 18, 2014

Change of focus on the Rohonc project

The Rohonc transcription has reached a point where I think it is "good enough for now". I have identified about 93% of the glyphs (including 100% of glyphs in the first 50 pages), corrected some problems with the order of lines, and identified those lines that are damaged either at the head or the tail.

If a solution is possible, I doubt it would rely on the 7% of glyphs that I haven't identified yet. Now I'm going to change my focus to word breaks, since I think the data is good enough to apply the formula I mentioned in my last post for identifying word breaks.

In the mean time, I have started a few other projects, so hopefully I will soon be able to post about some other interesting stuff in addition to the Rohonc Codex.

Wednesday, August 20, 2014

Word breaks in the Rohonc Codex

I have developed some really cool tools that work on the word level of a text, and identify things like the level of semantic content in words (to differentiate function words from content words), semantic similarity, and so forth.

Unfortunately, I can't use those tools on the Rohonc Codex yet because the transcription is on the level of the glyph, not the word.

But the codex gives us a few tiny clues to word breaks, and I think I have figured out how to leverage those to split the text into words.

The main clue is the hyphen, which looks like a double line. The hyphen at the end of a line tells us where a word break probably is not, and the absence of a hyphen tells us where a word break probably is. This allows us to build a small table of likely word heads and tails, which might be used to identify word breaks (where a break may be between a likely tail and head).

As a proof-of-concept, I took an English text with 47,975 words, removed all of the spaces and punctuation, then divided the text into lines of approximately 80 characters, breaking lines at word boundaries. I then measured the frequency of every segment that started or ended a line of text, and wrote a scoring function that looks like this:


In this equation, b(x, y) is the "break score" of a point between strings x and y, the variable tx represents the number of times that string x is found at the end of a line, and hy is the number of times that string y is found at the head of the line.

If the break score crosses a certain threshold, then I insert a word break between strings x and y.

There are a few parameters that govern how you actually implement this. The first is the length of the strings x and y, and the second is the threshold to use. Experimenting with my English text, I found the optimal string length to be 3, and the optimal threshold to be 10. With these parameters, I was able to correctly identify 78% of word breaks in the English text, with only one incorrect word break for every 10 correct word breaks.

Before I apply this to the Rohonc Codex, I think I will put some more effort into cleaning up the transcription. In order for the word-breaking algorithm to work efficiently, I will need to identify lines that are damaged and possibly missing glyphs on one or the other end.

In the meantime, I am wondering what it would look like if we apply this to the Voynich manuscript. It would be interesting if the apparent word breaks in the VM don't strongly correlate with the word breaks identified by this approach.