Showing posts with label Linguistics. Show all posts
Showing posts with label Linguistics. Show all posts

Saturday, July 24, 2021

Linguistic Information from Voynich 81R

Page 81R of the Voynich Manuscript has a block of text with an interesting property that is different from other text in the VM. While most lines of text in the VM continue to a page margin or the boundary of an image, the text on 81R is ragged on the right side. In other texts, both printed and manuscript, this type of raggedness can be a property of poetry, wherein the breaks between lines are guided by metrical considerations rather than the need to use space on the page efficiently. Nick Pelling has a post that digs into this page, and he notes that the poem-like layout of this page was observed by Gabriel Landini on the Voynich mailing list in 1996.

So, if 81R contains a poem, then what kind of information could we derive from it? 

Generally speaking, a line of poetry is broken down into feet, and feet have some relationship to syllables, though the exact nature of that relationship varies. In Latin and Greek dactylic hexameter, for example, a foot is made of two poetically long syllables (a spondee) or else a long syllable and two short ones (a dactyl), and there are six feet per line. In iambic pentameter a foot is made of one unstressed and one stressed syllable (an iamb) and there are five feet per line. Other styles of poetry use other definitions of feet and other numbers of feet per line.

Whatever definition there is to a foot and a line, however, there is going to be some natural relationship between the length of the line and the number of syllables in it. For a given language and a given metrical form, that will lead to a certain average number of words per line, with a certain standard deviation.

These values are different for different languages and metrical forms. In the graph below, I have taken multiple 31-line samples from five epic poems and graphed the average number of words per line (x axis) against the standard deviation in number of words per line (y axis).


In the graph above, you can see that each of these epic poems has a different average number of words per line. Chaucer is by far the highest, while the Serbian epic poem Strahinja Banović is at the extreme other end.

If the number of words in a line of Voynich text is equal to the number of words in a line of the underlying plain text, and the text on 81R is a poem, then where does it fall on this graph?

There is not universal agreement on where all of the wordbreaks are on 81R, so we have a range of answers, but it is a relatively narrow range, and the answer is relatively clear. Among the five sample epic poems, the most similar in this respect is the Aeneid. The three red bubbles in the graph below demonstrate the range of values for page 81R, and the blue bubbles are the sample values from the Aeneid.

The next most similar poem is the Anglo-Norman Voyage de Brendan, the right edge of which touches the left edge of the Voynich range:

This suggests the possibility that 81R is written in Latin dactylic hexameter, or else possibly something like the Anglo-Norman octosyllables of Voyage de Brendan.

The argument for Latin dactylic hexameter is strengthened over something like Old French by the fact that there are 31 lines on 81R. Old French poetry (like Middle English) was built on rhyming couplets, and to have an odd number of lines would mean having a line dangling at the end with no rhyme.

Of course, the VM is not a simple substitution cipher, and it's always possible that a Voynichese word does not correspond to a plaintext word, but this is a direction I will hopefully expand on more in my next post.

Wednesday, March 4, 2015

The Meroitic Romance

I have recently been reading Aethiopica, a novel written in Greek by a Phoenician named Heliodorus, from the city of Emesa. (I am reading Wright's 19th century revision of Thomas Underdowne's 1587 translation.)

The novel was probably written around 200 - 300 AD, but it is set in Achaemenid Egypt, when Egypt was a satrapy of the Persian empire (between 525 - 332 BC).

One of the things I love about this text is that the realities of language and ethnicity are everywhere. Greeks in Egypt must find another Greek (often a captured Greek slave) to act as translator. The wife of the Persian governor of Memphis, though she understands Greek, cannot speak it, so she too needs a translator to communicate with the Greeks. Phoenicians on trading expeditions are found in Delphi, and recognized at other islands in the Mediterranean by their huge and well-built ships.

One thread of the complex plot involves a mysterious letter written in an unknown language. This is in the possession of the main female character of the story, Chariclea, the adopted daughter of a Greek priest of Apollo. Among the belongings he received when he adopted her is a letter written on cloth that is neither in Greek nor Egyptian. This mysterious letter is finally read by a visiting Egyptian priest, who says:
"It was in Ethiopian letters, not the common sort, but such as their princes use, like to those that the Egyptians call hieratic."
When Heliodorus tells us that they are "Ethiopian letters", however, he is not referring to the old Ge'ez abjad. Indeed, the Ge'ez letters do not look much like hieratic:

Ge'ez abjad, from Wikipedia


A piece of one of the Chester-Beatty papyri in hieratic

In fact, when Heliodorus says Ethiopia, he means the kingdom of Kush, and the writing system he is referring to must be the cursive form of Meroitic:

A Meroitic ostracon, from the Unesco website.

Meroitic is not yet fully deciphered, but the script is known, and it was in use for about 800 years from 300 BC - 600 AD. Its inclusion in the Aethiopica is probably a little anachronistic, but it is a sin that can easily be forgiven in such a well-written story from so long ago.

Monday, April 21, 2014

Rohoncian looks like a case-marking language with prepositions

The computer-assisted transcription of the Rohonc codex is about 61% done. So far, I'm resisting the urge to do any kind of statistical analysis because the results will almost certainly be skewed by the fact that my glyph recognition algorithm has a harder time with some glyphs than others.

However, I came across something today that I thought was interesting. Throughout the codex, there is a sequence of four glyphs that is usually written with a halo over it, as follows:

Co D Co D

Starting from this post, I'll use my provisional transcription system, so I don't have to talk about "the glyph that looks like a triangle", and so forth.  So this sequence is transcribed as Co D Co D.

I think the odds are good that this is a noun, or a proper noun, but of course it is not clear what it is. It could be an epithet of God, the Holy Spirit, or even an abstract noun like Grace, but for the sake of convenience I'll call it "the holy noun". At first blush, it looks like a reduplicated stem, but there are a few cases where it appears to be inflected.

In an earlier post, I suggested that the simple line bending to the left was a preposition line "on". I now transcribe this glyph as L (for "left"), and in the following sequence you can see two cases where the holy noun is preceded by L, and at the same time the ending of the sequence changes:

L C D C I Ix Hk L C D C I

It may or may not be important, but in this case the glyph Co is replaced by C. But since the only difference between the two is a small loop at the top, it is possible that they are allographs of the same grapheme.

More important is the suffix. It looks like the (tentatively) nominative suffix D becomes I when the noun is prefixed by the preposition L.

Compare also the following:

O Co D C D C

It seems fairly clear that this is the same holy noun (marked with a halo, as always), but it has an added C suffix after the D. This may or may not be related to the O that precedes the whole word.

If these sequences really do show prepositions and case-marking, then it narrows the field of candidate languages quite a bit. We would be looking for a language with prepositions and at least three cases. Here is the rough paradigm of a noun in -D:

Possible Nominative: stem + D
Oblique A: stem + I
Oblique B: stem + D C

Monday, February 3, 2014

The proper names of the saha islands

In a couple of recent posts, I've been gnawing on the names of some islands at the mouth of the Tumen river, where the word saha appears to mean "island".

Initially I was working from European maps, which alternated between saba and saha, and I had thought that the word must be saba.

However, today I remembered a volume at the Bibliothèque national de France containing a long list of Manchu place names. I skimmed through it, and managed to find the page shown below, which now allows me to give the proper names for the islands, as transcribed from Manchu script.

The islands listed on Danville's map are (from South to North) Taitou saha, Siské loun, Tayam ou saha, Sarbatchou saba, Mama saba, and Youanga toun. The names, as transcribed from the page below in Manchu script, are Daidu saha, Sishe tun, Dayanggū saha, Sarbacu saha, Mama saha and Yohangga tun (not shown on this page, but on a later page).

Also listed on this page is a river called Fiya bira (bira meaning "river"), which bears a passing resemblance to the ethnic name Fiyaka.


Friday, January 31, 2014

A possible candidate for the saha-islanders

In my last post I remarked on the word saha, apparently meaning "island" in the names of some islands just north of the mouth of the Tumen river.  As far as I can tell, this is not a word in a known language of the area.

In this post, I will propose two possible candidate ethnic groups for the speakers of that language. The following text is a Manchu-language description of groups of people called Fiyaka and Kiyakara who brought tribute to the Qing:
fiyaka, fiyaka sunggari ulai dergi ergi ten i bade bi, mederi tun i jakarame son son i tembi. nimaha butame gurgu buthašame banjimbi. hahasi hehesi gemu indahūn sukū be etuku arafi etumbi. juwari forgon de nimaha sukū i arambi. banin doksin becunure de amuran. tucire dosire de kemuni jeyengge agūra gaifi yabumbi. aniyadari seke jafambi.
kiyakara, kiyakara huncun mederi, jai fucin yose i jergi birai biturame son son i tembi. haha hehe gemu ferten de muheren etufi jurhun isire menggun teišun i araha niyalma be miyamigan obume tuhebumbi. hahasi buhū i sukū be mahala arambi. bosoi etuku etumbi. bethe niohušulembi. hehesi funiyehe be tuhebume, sifikū sifirakū, adasun de halai hacin i šeoleme wangnambi. boo ūlen jahūdai weihu be gemu alan i weilembi. ese asu baitalame bahanarakū. nimaha šakarame gurgušeme banjimbi. banitai heolen sula iktambume asaraha hacin akū. ceni ba i ici gisurere be kiyakaratambi sembi. aniyadari seke jafambi.
"The Fiyaka. The Fiyaka are in the high places on the east side of the Sunggari river. They are scattered along the island(s) of the sea. They make their living fishing and hunting. Men and women all make and wear clothing of dog skins. In the summer they make them from fish skins. By nature they are cruel and they love to fight. When they are out and about they walk carrying bladed spears. Every year they bring sable as tribute.
"The Kiyakara. The Kiyakara are scattered along the Hunchun Sea, and along such rivers as Fucin and Yose. The men and women all wear rings in their noses, and hang figurines as ornaments made of silver and copper as much as an inch long. The men make hats from deerskin, wear cloth, and go barefoot. The women let their hair down, and do not wear hairpins. They embroider their lapels with all kinds of different designs. They make every kind of house and boat from birch bark. They do not know how to use nets. They make their living by spearing fish and hunting. They are naturally lazy and idle, and do not customarily accumulate and set aside stores. The speech of their land is called kiyakaratambi. Every year they bring sable as tribute."
The Fiyaka and Kiyakara occupied similar territories, as far as I can tell. Both groups are now said to have been Tungusic, but I think the evidence for this is based largely on the fact that Tungusic speakers are known to have lived in the areas they inhabited. The same section of the tribute records also mentions Nanai (heje) and Udeghe (nadan hala), as well as non-Tungusic Ainu (guye or kuye) and Nivkh (kilen).

Thursday, January 30, 2014

In what language does saha mean "island"?

This evening, I have been poring over Danville's beautiful 18th century map of Korea and Manchuria, looking at the tangled area where Korean toponyms give way to Manchu/Jurchen (and other non-Korean) toponyms.

I started out hunting for two places called (in Manchu) ehe kuren and gūnaka kuren, which I expect to find on Danville's map with a spelling like *eghe couren, *counaca couren. I have not found them yet, but I have found something else that is interesting.

All along the Korean coast, small islands are given names ending in tao, which no doubt corresponds to Chinese 島, "island" (Mandarin dǎo).

Some Korean islands with names ending in "tao"

Along the Japanese coast, of course, we have small islands whose names end in sima, corresponding to the Japanese word shima, "island".

Some Japanese islands with names ending in "sima"

But at the mouth of the Tumen river, there is something I did not expect. Some of the islands have names ending in toun, which must be Manchu tun, "island", and some have names in loun, which I am certain is a scribal error for toun. But others have names in saba and saha, which I don't recognize.

A mix of islands with names ending in "toun/loun" and "saba/saha"

I expect saha is a scribal error for saba, because I see the same error in the name of one of the tributaries of the Tumen, called Cahari at one point and Cabari at another. It is more likely that the correct form is saba because intervocalic -h- is rare in these place names, being usually represented by -kh- or -gh-. But in what language does saba mean island?

Note 3 February 2014: It turns out I was wrong. The word is saha, as I note in a later post. I've corrected the remainder of this post.

The major language families of the area (broadly speaking) are Tungusic, Japonic, Korean, Nivkh, Ainu, Mongolic and Chinese. Of all of these, the only thing I have found so far that looks similar is Korean seom 섬, "island", which really looks closer to Japanese shima, as far as I know.

It looks like there is a somewhat large island called mama saha, and a smaller one called sarbatchou saha. Given similar island names throughout the world, it would not surprise me if these are "mother island" and "daughter island", so maybe that is a place to start.

In Manchu, sargan means "woman" or "wife", and sargan jui means daughter. However, Tsintsius lists this as a Manchu morpheme only, not attested in other Tungusic languages, so it falls into what I call the "enigmatic vocabulary" of Manchu: common words with no known origin.

I have often felt that the enigmatic vocabulary of Manchu provides evidence of close contact between the ancestors of the Jurchens and speakers of an otherwise unattested language. Under this theory, words like sargan would have been loaned into Manchu from the other language, which then disappeared. If sarbatchou saha really does mean "daughter island", perhaps it comes from the same source language, or a close relative.

A close study of the place names in Danville's map may lead to more interesting discoveries.

Sunday, January 19, 2014

Secret sign

I was walking down a steep mountain path in Sichuan with a local guide, paying rapt attention as he told me stories about the area. There were graves in the hillside, he said, and as a troublesome young man he once looked into them, and was terrified by the corpses. Another time he lost his favorite horse, who slipped on the path and fell to his death in a deep ravine. That complex in the valley was a prison, where he had spent some time.

Let's get together in the village later in the day, he said finally. But let's lose the Chinese guy. I don't trust him.

Indeed, I had been curious about the soft, overweight Chinese man in our party. He did not seem physically suited to a three-day horse ride, and he seemed to prefer reading stories on his cell phone to enjoying the dramatic scenery of the Sichuan mountains. Why was he there? My guide seemed to find it suspicious.

If anyone had been listening in, they would have been completely unaware of our conversation on the matter. This is because my guide was deaf, and we were communicating in Chinese Sign Language, of which I had managed to learn a fair amount over the prior three days.

In a previous post, I mentioned some qualities of a good secret language. Here, let me extol the virtues of sign language as an effective means of secret communication in the 21st century.

A secret language is, roughly speaking, a substitution cipher that operates on the level of morphology and grammar. Experience teaches us that unknown languages are difficult to decipher, so as long as the "key" remains a secret, the language remains relatively secure. The "key", in this case, is the combination of lexicon and grammar.

As a cryptographic system, secret languages are terrible. The key is difficult to transmit, and once broken, a new key must be laboriously created and transmitted. However, the great saving grace of secret languages in the 21st century is that encryption can take place entirely within the only device that remains free of malware: the human brain.

In order to remain secure, however, encryption must remain within the human brain. One of the significant weaknesses of secret languages in the era of the surveillance state is that users may be tempted to store or transmit the lexicon and grammar in an electronic form that may be intercepted and compromised. Another weakness is that keywords in the secret language may be distinctive enough that secret messages may be easily identified and used for traffic analysis.

A secret sign language is more secure on both of these counts. First, the key is actually difficult to store in writing, and is most naturally communicated person-to-person. Second, the easiest way to transmit a message over the internet is by video, which requires much more extensive and complex analysis even to pick out the existence of the secret communication.

Today's surveillance states have vast means at their disposal, and can easily out-spend and out-compute most of their adversaries. For the time being, however, there are a few faculties of the human mind that remain out of the reach of conventional computation. A secret sign language takes advantage of many of these capabilities, at a relatively cheap cost.

Friday, November 8, 2013

FST via RDBMS

I was stuck in an airport a couple days ago with nothing to do, so I started a little project to pass the time. The challenge is this: Using a conventional database and a conventional programming language, what is the fastest program I can write that can take an inflected word in some given language and identify the root and the inflection(s)?

There are two ends of the spectrum in solutions. On one end, you have a "fat" solution involving a massive database containing every possible inflected form of every word in the language. On the other end, you have a "lean" solution with a smaller database of stems and inflectional information, and some complex code to use the two in combination.

The fat solution has the advantage of using mature indexing technology to efficiently search the massive database for the subject term.  Supposing you have a binary tree index, the cost of executing a query with the fat solution might be:

Tfat = Q + N log2 (si)

where Q is the fixed cost of opening a query to the database, N is the incremental cost of checking one node in the binary tree, and log2 (si) is the depth of the binary tree, based on the number of stems (s) and inflections (i).

However, for agglutinative languages this is a nightmare. Consider the standard Uyghur expression used when meeting someone new:

tonušqanliqimizdin xošalmen

The stem of the first word is ton, and the rest is all inflection (uš - qan - liq - imiz - din). A database containing all inflected forms of Uyghur verbs would be huge.

The leanest solution searches the stems and inflections separately. Since the inflections are a fixed set, it makes sense to factor them out first, then do a simple search of the stems.  The speed of the lean solution is the time required to query for all of the inflectional data (Q + Ni), the time required to parse all possible inflections (P), and the time required to query roots (Q + N log2 s):

Tlean = (Q + Ni) + P + (Q + N log2 s)

which can be written as

Tlean = (Q + Ni) + P + Tfat - N log2 i

In order for the lean solution to beat the fat solution, this has to be true:

Q + Ni + P < N log2 i

That simply isn't doable, because Q, P and N are positive, and i > log2 i.  In order to get the lean solution to work, we've got to pre-fetch the inflectional data. In fact, we could write a routine to use the inflectional data to build a parser, in which case we're down to this:

Tlean = P + Tfat - N log2 i

And now all we need is to make this work:

P < N log2 i

The fastest parser would probably be a finite state transducer. It seems to me you would end up with the FST traversing about the same number of states as the nodes traversed by the index search, but the FST has the advantage of running in memory, while the index search would have to deal with some of the necessary evils of relational database management (like query preparation, I/O buffering, and so forth) that are hiding in the value N.

Going back to the challenge requirement that this has to be implemented in a conventional programming language, the best marriage between conventional programming and an FST is probably found in regular expression processing, so that would probably be the tool to use for the parser.

Tuesday, October 1, 2013

I guess this is Voynich month

I hate to get stuck on things, but I feel like I have to exhaust this idea that the Voynich gallows letters could be stressed vowels.  I'll call this the GAV (gallows-as-vowels) hypothesis.

I was in a hurry with my last few posts, and I'm not someone who is extremely familiar with the VM, so I overlooked something that should really be incorporated into the proposal.  First, the gallows letters have an alternate form called "platform gallows" where they appear in ligature with another grapheme represented in EVA as <ch>.  Since it is not uncommon in Latinate scripts to form ligatures of vowels (like œ, æ) and vowels are by far the most common carriers of suprasegmental markers (like á, ä, ā, â) it seems reasonable to count <ch> and the apparent variant <sh> with the vowels somehow.

So the GAV thesis is this:
The [main] underlying language of the VM is a language with a bias towards stress on the first syllable. The words of the text are abbreviated by writing primarily the consonants, and excluding most unstressed vowels. The vowels that are retained are represented by gallows letters, by <ch> and <sh>, and by the ligatures of <ch> with the gallows letters.
This system of abbreviation would introduce a certain amount of collision, where the abbreviated forms of different words would be written the same.  However, many ambiguous forms could be understood from context, and there could also be a mechanism to avoid collisions for frequent or important words.

Interestingly, if this is really how it works, it suggests that the alphabet was deliberately designed to mislead the uninitiated by making the vowels look like consonants by giving them large, imposing shapes, while some consonants were given shapes more like vowels.

So, where do we go from here?  I think we go to syllable structure.

It has long been known that Voynich words adhere to some kind of structure.  Under the GAV thesis, much of this structure will correspond directly to the structure of the stressed syllables. If we look at the multiliteral clusters that appear before the *vowel, there is a set of common permissible clusters in <q, qo, qol, o, ol, l>, together with a set of largely impermissible clusters <*oq, *oql, *lq, *ql, *lo, *loq>.

If the language were English, the permissible clusters could represent (for example) [s, st, str, t, tr, r], while the impermissible ones could represent [*ts, *tsr, *rs, *sr, *rt, *rts].  Other solutions are possible, both in English and in other languages.

So, which languages should we look at? The manuscript is supposed to have been sent to Athanasius Kirscher from Prague, so we would probably start there.  The languages of Bohemia included Bohemian (Czech), which stresses the first syllable of words; and German, which tends to stress the first syllable.  Hungary is not far away, but while Hungarian stresses the first syllable, it doesn't tend to have as many word-initial consonant clusters as would be required under the GAV theory.

Lastly, we should probably throw in English just because Johannes Marci told Kirscher that Ferdinand III's Bohemian tutor thought the book had come from Francis Bacon.

So, given that <ol> is a common word, as well as being a member of the permissible consonant cluster series above, we should probably assume it is unstressed and written without its vowel.  Among these languages, where can we find a permissible consonant cluster series like <q, qo, qol, o, ol, l> where the term <ol> in the series could plausibly be a common word?

Tuesday, September 24, 2013

What would it mean if the Voynich gallows letters are vowels?

In my last post, I made a proposal about vowels, and inadvertently revealed that I am not a real Voynichologist, because I referred to [p,f,t,k] as "tall, loopy letters" instead of using their conventional name: gallows letters.

First, consider the distribution of the gallows letters within words.  The following graph shows the relative frequency of gallows letters compared to other letters in the first ten letters of Voynich words:


Gallows letters are heavily represented at the beginnings of words, and scarcely represented later in words.  If they represent vowels, then what does this imply?  Perhaps the language has syllable stress, and the stress normally falls on the first syllable of a word, and only the stressed vowel is written.

That could not be the whole picture, however.  In English, a language with a bias towards stressing the first syllable, the relative frequency of the vowels compared to other phonemes in the first ten phonemes looks like this (data from the Carnegie Mellon pronunciation database):


As you can see, stressed vowels are slightly over parity with other phonemes as the second phoneme of a word in English.  To explain the gallows letters as stressed vowels, we would additionally have to assume some particular stressed vowel or vowels in the first syllable if no other vowel is written, and this would have to apply to about 4/5 words that are stressed on the first syllable.

Monday, September 23, 2013

I couldn't resist: The Voynich Vowels

OK, I didn't mean to get started down this route because I'm a little overworked, but I ran my contextual distance analysis on the Voynich graphemes (in EVA transcription).  Based on that, here is my best guess at the Voynich vowels.

The following image shows a map of the similarity of the graphemes in the Voynich Manuscript.  Like the King James genesis, we have a small island of five graphemes which occur with the word boundary (#).  I will call these the *vowels (note the asterisk, which is my caveat that this is all hypothetical).


The *vowels, in EVA transcription, are the letters z, t, k, f and p.  The Voynich graphemes represented here are:


There is an obvious graphemic similarity here:  The tall, loopy letters are all *vowels, as is the z, which might be called a small, loopy letter.

Interestingly, the closest contextual similarities here are between the graphemes that look the most similar:

sim(t, k) = 0.88
sim(f, p) = 0.87

By contrast, the strongest similarities between any two vowels in Latin is sim(e, i) = 0.77, while in English it is sim(a, o) = 0.59.

If the *vowels are indeed the Voynich vowels, then I propose that we are looking at a script with only three vowels, and the EVA [t,k] and [f,p] are merely graphemic variants of each other.

Is it possible that the underlying orthography is based on something like the way Hebrew is used to write Yiddish, using the [א,ו,י] to represent vowels?

In fact, I'll go a step further, just so I can claim credit if this turns out to be true:  Given my earlier feeling that this is a polyglot or macaronic text, is it possible that the underlying text is in Hebrew and Yiddish (or another language of the Jewish diaspora, like Ladino)?

How would you use semantic similarities for decipherment?

A couple weeks ago, I started taking an online course in cryptography taught by Dan Boneh from Stanford on Coursera.  (It's excellent, by the way.)  I've been a little slow to post anything new, but I wanted to incompletely finish a thought.

Why should we care about mapping semantic relationships in a text?  Well, I think it could be a valuable tool for deciphering an unknown language.

First, what I've been calling "semantic similarity" is really contextual similarity, but it happens to coincide with semantic similarity in content words (content-bearing nouns, verbs, adjectives, adverbs). In function words, it's really functional similarity.  In phonemes (which I don't think I have mentioned before) it correlates with phonemic similarity.

Here's an example of a map generated from the alphabetic letters in the book of Genesis in King James English:


The most obvious feature of this map is that all of the vowels (and the word-boundary #) are off on their own, separated from all of the consonants.  If we had an alphabetic text in an unknown script in which vowels and consonants were written, we could probably identify the vowels.

There are finer distinctions that I may get into one day when I have more time, but the main point here is that contextual similarities of this type measure something valuable from a decipherment perspective. I believe it should be possible to get the general shape of the phonology of an alphabetic writing system, to separate out prepositions or postpositions (if they exist), and to eventually distinguish parts of speech.

Friday, September 13, 2013

Visualizing the semantic relationships within a text

In the last few posts, I had some images that were supposed to give a rough idea of the semantic relationships within a text.  Over the last few days I have been rewriting my lexical analysis tools in C# (the old ones were in C++ and Tcl), and that has given me a chance to play with some new ways of looking at the data.

When I depict this data in two dimensions, I'm really showing something like the shadow of a multi-dimensional object, and I have to catch that object at the right angle to get a meaningful shadow.  In the following images, I have imagined the tokens in the text as automata that are bound by certain rules, but otherwise behaving randomly, in an attempt to produce an organic image of the data.

For example, each point of this image represents a word in the lexicon of the King James version of the book of Genesis.  The words all started at random positions in space, but were made to iteratively organize themselves so that their physical distances from other words roughly reflected the semantic distance between them.  After running for an hour, the process produced the following image:


There are clearly some groups forming within the universe of words.  Since this process is iterative, the longer it runs the more ordered the data should become (up to a point)...but I haven't had the patience to run it for more than an hour.

The same process applied to the Voynich manuscript produces something smoother, as you can see below.  However, the VM has a much broader vocabulary, so I estimate I would need about nine hours of processing to arrive at the same degree of evolution as I have in the image above.


Reflecting on that, I thought I would try a new algorithm where words are attracted to each other in proportion to their similarity.  The result was not interesting enough to show here, since the words rapidly collapsed into a small number of points.  I am certain that this data would be interesting to analyze, but it is not interesting to look at.

Thinking about the fractal nature of this collapse, I thought I would use similarity data to model something like the growth of a plant.  In the following image, I have taken the lexicon of the King James Genesis again and used it to grow a vine.  I started with one arbitrarily chosen token, then added the others by joining them to whatever existing point was semantically nearest.  Each time, I tried up to ten times to find a random angle of growth that did not overlap another existing line, before finally allowing one line to overlap others.

I am quite pleased with the result here, since it shows what I have always intuitively felt about the data--that there are semantic clusters of words in the lexicon.


The same approach applied to the Voynich manuscript produces a denser image, due to the greater breadth of the vocabulary:


But how does this compare to random data?  To answer this question, I processed a random permutation of the Voynich manuscript, so the text would have the same word frequency and lexicon as the Voynich manuscript, but any semantic context would be destroyed.  Here is the result:


Intuitively, I feel that the random data is spread more evenly than the Voynich data, but to get beyond intuition I need a metric I can use to measure it.

Good night.

Tuesday, September 3, 2013

Theory about pre-human language

I have a theory about the course of the evolution of human language that is too long to put into a blog post.  I have given it its own page, called Pre-Human Language.

Sunday, September 1, 2013

The Ants and the Cricket

I took a shot at translating one of Aesop's fables into the Grande Ronde dialect of Chinuk Wawa, as documented in Henry Zenk's excellent dictionary.  Here it is. (If any Wawa speakers happen to come across this, I am happy to hear feedback and corrections).

The original was "The Ants and the Grasshopper".  I made it "The Ants and the Cricket", since a cricket is called a "useless bird" (kʰəltəs-kələkələ), and the cricket sings, while the grasshopper doesn't.

t’siqʰwaʔ pi kʰəltəs-kələkələ
The ants and the cricket

kʰul-iliʔi ixt san t’siqʰwaʔ ɬas-munk-tlay məkʰmək ikta wam-iliʔi ɬas-munk-iskam.
One day in the winter, the ants were drying food which they had gathered in the summer.

kʰəltəs-kələkələ ya-chaku miməlust-ulu.
Cricket became extremely hungry.

ya-ɬatwa kʰapa t’siqʰwaʔ ɬas-haws, ya-aləksh məkʰmək.
He went to the ants' house, and he begged for food.

uk t’siqʰwaʔ ɬas-wawa, "pus-ikta wam-iliʔi wik ma-munk-iskam məkʰmək?"
The ants said to him, "Why did you not gather food in the summer?"

kʰəltəs-kələkələ ya-wawa, "wik na-t’uʔan lili pus na-munk-iskam məkʰmək.
The cricket said, "I didn't have time to gather food.

kʰanawi wam-iliʔi na-shati shati."
The whole summer long I sang and sang."

ɬas-munk-hihi-yaka, wawa, "pus kʰanawi wam-iliʔi ma-hihi pi shati, aɬqi kʰanawi kʰul-iliʔi ma-tanis pi ulu."
They laughed at him, and said, "If you laugh and sing all summer, then you dance and hunger all winter."

Thursday, August 1, 2013

A good secret language

Just for fun, I thought I would say something about what could make a good secret language.

A secret language would be "good" to the extent that it met its requirements.  The obvious first requirement of a secret language is that it protect the communication that it encodes, so it should be different from known public languages.  Beyond mere strangeness, however, there are a number of supporting characteristics that a secret language might have.

1. Learnability.  In order to be used, a secret language must be learned.  The easier it is to learn, the more quickly it may be used.  Learnability is improved by:

1.1 Regularity.  Irregular forms make languages harder to learn, so a language with regular rules is easier to learn.

1.2 Phonological and Syntactic Familiarity.  If speakers of a secret language share a common public language, then the secret language will be easier to learn if it shares the phonology and syntax of the public language.  This occurs in Anglo-Romani and Thieves' Cant.

1.3 Redundancy.  A natural language makes relatively efficient use of the available phonological space for words.  For example, in Chinese, there is at least one morpheme for almost every possible Chinese monosyllable.  However, since the secret language is learned as a second language, it will benefit from making morphemes more distinct from each other than you would normally find in a natural language.  This occurs in Chinook Jargon.

2. Extensibility.  The core lexicon for a secret language may be relatively small, with a powerful derivational morphology to extend the lexicon over its intended domain of communication.  This makes the language easier to learn, because there are fewer morphemes to memorize, but also makes it possible to invent new terminology on the spot with a high likelihood that you will be understood.  I don't know of a secret language "in the wild" that does this.

3. Obfuscation. Some phonological features of natural languages could hypothetically be used for obfuscation.  Nearly all of these would come at the expense of lost redundancy.  For example:

3.1 Avoiding Labials.  If you wanted a secret language that could not be lip-read, you would avoid labials (including rounded vowels).  I have read a couple descriptions of the Chinook language that say Chinook speakers would hardly move their lips at all.

3.2 Avoiding Voiceless Fricatives, Affricates and Aspirates.  If you wanted a secret language whose sounds would not carry far, you might avoid sounds that are accompanied by high frequency formants, which carry farther.

Sunday, July 21, 2013

Secret Languages

I love the idea of secret languages.  It started out as a fascination with foreign writing systems when I was a kid, which turned into a love of ciphers and codes.  Around age 10 or so, I decided I wanted to create a code you could speak, so I started with a simple cipher where consonants were shifted up the alphabet by one letter, and vowels likewise shifted up the list of vowels:
O jewi e tidsiv mephaehi 
I have a secret language
This kind of code was somewhat speakable, but I realized it was also creating strange and unusual consonant clusters (as in the word tvsuph for "strong").  I gradually started dividing the sounds of English into continuants and stops, then I discovered place of articulation, and manner of articulation. Eventually I had a fairly sophisticated, relatively speakable code that first required the English plaintext to be represented phonetically, then shifted around in terms of place of articulation:

el rɛɣ ə θutʃjɪp wɛndʒhʊdz 
I have a secret language
Then, one day, I realized you could solve the whole pronunciation problem by substituting morphemes, and my brother and I started to create an actual secret language.  It went through many revisions, but in the latest form that the language has in my mind, you can say:
tan kɪp tʃ tollani namrɪl 
I have a secret language
Though my brother's dialect is different from mine, so to him I would say:
tan kɪp tʃ namrɪl koji ekaj tol la 
I have a language that other people do not know
Over the years of playing with this, I have landed on a kind of formula for effective secret languages.  I have run out of time for the moment, but an upcoming post will be about the formula for an effective, easy secret language.

Friday, July 19, 2013

渤海 Language

There was a kingdom called 渤海 in what is now North Korea and North-Eastern China.  For nationalistic reasons, certain modern nations are keen to assert that the people of that kingdom belonged to their own ethnic group and spoke their own language. The system of romanization you use has intense political ramifications. (Just read the talk pages on Wikipedia for this and related kingdoms.)

In order to sidestep this debate, I will use Chinese characters instead of contested historical names throughout this post.

No texts in the 渤海 language survive, so all we can do is speculate based on historical sources.  In fact, it appears that the 渤海 may have used the Chinese language as their literary language, based on a passage from the history of the Jurchen Jīn dynasty:
乙未,诏百官诰命,女直、契丹、汉人各用本字,渤海同汉人。
On the yiwei day, [the emperor] issued edicts to the hundreds of officials.  For the Jurchens, Khitans and Chinese, he used their own scripts; For the 渤海, [he used] the same as the Chinese.
Ethnically, the Jurchens seem to have believed that they were related to the 渤海, since the Taizu of the Jīn dynasty promulgated an edict saying:
女直、渤海本同一家。
The Jurchens and 渤海 are originally one family.
Indeed, the history of the Jurchen Jīn says:
粟末靺鞨始附高麗,姓大氏。李績破高麗,粟末靺鞨保東牟山。後爲渤海,稱王,傳十餘世。有文字、禮樂、官府、制度。有五京、十五府、六十二州。 
The 粟末靺鞨 allied themselves to 高麗, and took the surname Dashi.  After [the Tang official named] Li Ji destroyed 高麗, the 粟末靺鞨 guarded over Dongmo Hill.  Later, they became the 渤海, were called kings, and for ten generations had writing, laws, music, official courts, systems, customs, five capitals, fifteen prefectures, and sixty-two districts.
The 粟末靺鞨 here are one of the seven tribes of 靺鞨 from whom the Jurchens held themselves to be descended.  The characters 粟末 are reconstructed as *sok-mat in OCM (Minimal Old Chinese), and represent an ancient name for the Sunggari river (now called Songhua).  I expect the river name was something like *sugmar, which became *sumgar through metathesis, *sunggar through assimilation.

If we assume that the collective name of the seven tribes, 靺鞨, was pronounced the same as the phonetics 末曷, then that gives is an OCM reconstruction of *mat-gât.  The Early Middle Chinese reconstruction of 渤海, for comparison, is *bət-xəy'.  It seems like it is not entirely impossible to say that the name 渤海, which appeared in the Táng, is a later form of the older 靺鞨, having undergone some sound changes.  The sound change from m > b is not well represented, though we do find it in the Manchu words for "monkey", monio and bonio.

The language of 渤海, if it was anything other than pre-Jurchen, should have left some loan words in Manchu.  Indeed, the Manchu language has a large store of enigmatic words whose etymologies can't be found in Tungusic, Mongolic or Chinese.  Much of these will no doubt be Khitan, but some of them may be 渤海.  Khitan itself is likely to have some 渤海 loans, as well.

Someone really needs to do a careful archeology of Manchu in order to make sense of the enigmatic vocabulary.  Among the words whose origins are indeterminate are many words containing palatal velars (represented by -giy-, -kiy-, -hiy- in the Manchu script), including the name of the Manchu royal clan, gioro.  The following relatively common words are also enigmatic: boigon, "family"; boo, "house"; bonio, "monkey"; abka, "heaven"; aga, "rain"; wece-, "to shamanize"; ganio, "supernatural".

Someone also needs to do an analysis of the pre-Chinese place names of the Changbaishan, among which there are a fair number with palatal velars.  I wonder if a careful linguistic archeology might not reveal the presence of an otherwise unattested substratum language in the area of the Yalu river and Changbaishan.

Friday, July 12, 2013

The Bulgar Calendar (done to death)

So I'm going to try to nail down my final interpretation of the Bulgar Calendar cycle.

Start with the Chatalar inscription, which has the year 821 in the Byzantine calendar, and sigor elem in the Bulgar calendar:

sigor, ox
elem, metal | white

Telets was defeated by the Byzantines in battle at Anchialus in 763.  If he reigned for three years, then somor altem, when he took the throne, must be on or after 760.  If altem = alem = elem, then that narrows it down to 760 or 761. As it happens, 761 was also an Ox year, and we know Ox was not somor, so we'll say he took power in 760:

somor, rat
altem, metal | white

Umor took the throne after Telets, so 763 or 764.  There are some caveats here: nomadic empires tended to choose their leaders by council (thus explaining why the different kings came from different clans), but also a new leader could not be chosen until the council had a chance to meet.  Not only that, but the first year of a reign was usually recorded as the first new year after taking power.  For now, let's be fuzzy about Umor's year:

dilom, rabbit (33%)  /  dragon (33%)  /  snake (33%)
toutom, water | black (33%)  /  wood | green (66%)

The reigns from Avitohol to Bezmer are partially legendary, so we can probably assume that the years and reigns make sense in the legendary context, even if we can't really tie them to historical events.  If there were two years from Gostun to Kurt, and Kurt took power in an Ox year (šegor večem), then Gostun must have taken power in a Boar year, so:

dohs, boar

Irnik's legendary 150-year rule would have started, then, in a Snake year (dilom tverim).  That settles the question about Umor's reign, which must have started in the Wood Snake year 765, so we can say:

dilom, snake
toutom, wood | green

If we assume Gostun's vtirem is a mistake for tvirem (which it should be based on the legendary reign lengths), then we know that tvirem and večem must be adjacent in the cycle of elements.  The cycle is Fire, Earth, Metal, Water, Wood, and we've already identified Metal and Wood, so the only adjacent unknowns are Fire and Earth.  That means:

tvirem, fire | red
večem, earth | yellow

And, by process of elimination, the remaining unknown element must be Water.

Tervel is fairly well-documented, being first mentioned in 704, and last mentioned by Theophanes the Confessor in 718 or 719.  If he ruled for 21 years, then there is no way that he could have taken power in a Water year (692-693 or 702-703).  Despite the fact that tvirem looks like a king name based on the format of the list, it seems more likely to be part of Tervel's year, and that the name of the king after Tervel is simply missing.  In that case, Tervel could have taken power in the Fire Rooster year of 697 and ruled for 21 years till 718, so:

tekučitem, rooster

Between Tervel and Telets (718-760) there are around 42 years.  This is best accounted for by assuming that there was no ruler between Tervel and Sevar.  Sevar took power in a Metal year, so either the Monkey or Rooster years of 720-721.  Since we already have Rooster, that leaves:

toh, monkey

Kormisosh must have taken power in the Fire Ox year (šegor tvirem) 737, accommodating the 15 years of the reign of Sevar and a couple years of interregnum.  From 737 till 760 there are 23 years to accommodate the 17-year reign of Kormisosh and the 7-year reign of Vinekh.  There is no way to read Vinekh's year as a šegor alem year, since those fell on 701 and 761.  Reluctantly, we must follow the text, and say that imaše goralem is most likely the Water Dragon year of 752

imaše, dragon
goralem, water | black

That leaves the vereni alem of Asparukh's reign.  Asparukh reached the Danube around 674-678, and Tervel took power in 697.  If we believe the 61-year reign, then this should either be the Tiger or Rabbit years of 630-631.

Notes added 10/29/2013:

Some of these words have Turkic reflexes (definitions below taken from Clauson):

altem, alem, elem, metal | white; compare Old Turkic altu:n, gold
dohs, boar; compare Old Turkic toŋuz, 'pig'
dilom, snake; compare Old Turkic yıla:n (< dıla:n), 'snake'
tekučitem, rooster; compare Old Turkic takı:ǧu:, 'domestic fowl'

Bulgar Calendar (revisited)

Always go back to the primary source.

I took a look at the Uvarov Transcript text of the Nominalia of the Bulgar Kings, and found a couple of interesting discrepancies between it and the translation I used in my last post.

To me, it looks like there is a king named Tvirem between Tervel and Sevar, who reigned for 28 years, and his year was dvanšehtem.  It's confusing, perhaps, because tvirem is a part of many of the year names, but it really does seem pretty clear.  Check it out:

Тервен. -к~а. лето. род ему Дуло. а лет ему текучитем.
твирем. ~(ки). лет. род ему Дуло. а род ему дваншехтем.
Севаръ. ~(еl). лет. род ему Дуло. а лет ему тохалтом.

I redid the list from the original:

Avitohol ruled 300 years from the year dilom tvirem
Irnik ruled 150 years from the year dilom tverim
Gostun ruled 2 years from the year dohs vtirem
Kurt ruled 60 years from the year šegor večem
Bezmer ruled 3 years from the year šegor vem
Asparuh ruled 61 years from the year vereni alem
Tervel ruled 21 years from the year tekučitem
Tvirem ruled 28 years from the year dvanšehtem
Sevar ruled 15 years from the year tohaltom
Kormisosh ruled 17 years from the year šegor tvirim
Vinekh ruled 7 years from the year imaše goralem
Telets ruled 3 years from the year somor altem
Umor ruled 40 days from the year dilom toutom

It seems like a huge coincidence that imaše goralem could also be parsed ima šegor alem.