\documentstyle[11pt]{article}

\textwidth13.5cm
\textheight20cm
\pagestyle{myheadings}
\markright{}
\markboth{}{}
\setlength{\topmargin}{-2cm}
\raggedbottom
\abovedisplayskip=3mm
\belowdisplayskip=3mm
\abovedisplayshortskip=0mm
\belowdisplayshortskip=2mm
\setlength{\baselineskip}{10pt}
\setlength{\evensidemargin}{0pt}
\setlength{\oddsidemargin}{0pt}
\setlength{\parskip}{0.13cm}

\begin{document}

\title{ Some Observations Regarding Corpora of some Indian Languages}
\author{
{\bf  Akshar Bharati}\\
{\bf  Rajeev Sangal }\\
 Satyam School of Applied Information Systems \\
Indian Institute of Information Technology, Hyderabad \\
\{sangal,smbsm\}@uohyd.ernet.in \\
\\
{\bf S. M. Bendre }\\
Department of Mathematics and Statistics \\
University of Hyderabad \\
\\
(In Proc. of Int. conf. on Knowledge Based Computer Systems (KBCS-98)\\
17-19 Dec. 1998, NCST, Mumbai.)\\
}
\date{}
\maketitle
\begin{abstract}

This paper presents some data from seven Indian languages, as
extracted from machine readable corpora. In particular, two kinds of
data are extracted: (1) Number of distinct high-frequency words in a
language needed to achieve a designated  P\%  coverage of a random
text. (2) Percentage of common words occuring in different languages.

The first data follows the expected trend that south Indian languages
have much larger number of distinct words because they are more
inflected than the north Indian languages and agglutinative in
nature. Based on randomly selected subsets of a given corpus, number of
distinct words needed to cover P\% of text is predicted using statistical 
methodology. The second data
shows that languages in adjacent regions have many more common words,
as expected. This information has been used to build
a software for detecting the language of an unknown text.

One of the aims of the paper is to introduce scholars to the possibilities of
using computers for extracting useful data about languages from
machine readable corpora and performing analysis using statistical techniques.
\end{abstract}


\section{Introduction}
In this paper, we present a study based on corpora of seven Indian languages
aimed at finding number of distinct words in a language and the number
of common words in any two languages.

A corpus is a collection of texts, which are usually carefully chosen
to be representative of the language. In this experiment, we have
used the DOE corpora. (These corpora were prepared
at the initiative of DOE through institutions which are acknowledged
at the end.) The size of each corpus is about 3 million words. Texts in
each corpus are categorized under literature (novels, short stories,
etc), mass media, social science, science, commerce, etc. The texts,
themselves, typically consist of a few pages randomly chosen from
publications during 1980-1990 in each of these categories.

Since we will be dealing with "word", first we give its definition. 
A word, for the purposes of this paper, is a sequence of alphabetic
characters, separated from other words by space or punctuation
marks. Thus 'usane' in Hindi is a single word because it is written
without any intervening space, whereas 'ladake ne' contains two
words. A word, therefore, for the purposes of this paper is defined
by the way it is written, without regard to the linguistic analysis
about its root, case ending, suffixes, etc.

In this study, data is extracted from corpora of 
different languages and a comparative
analysis is done relating to the number of distinct words needed to
cover a percentage of a random txt, and common words in use across
languages.



\section{Frequency of Words}

One important information that can be easily extracted from a corpus
is the list of distinct words that occur in it, together with how
many times they occur. Such a list is sorted in decreasing order of
frequency of occurrence, thus placing the most frequent word first,
and so on. For example, the first fifteen entries of the frequency
list for Hindi (prepared from the IITD Corpus) and Telugu (prepared from
the CIIL Corpus) are given in Table 1.

\begin{table}[hbt]\begin{center}
Table 1: Word-frequency tables for Hindi and Telugu. \\[0.2em]
\begin{tabular}{rlrl} \hline\multicolumn{2}{c}{ Hindi  Corpus }&\multicolumn{2}{c}{Telugu  Corpus} \\ \hline
   103216 & ke	 &       29548 & I \\
    87459 & hE	 &       18306 & A \\
    81128 & meM	 &       16429 & oVka \\
    70685 & kI &  9886 & kUdA \\
    52507 & se &  9289 & ani \\
    51630 & Ora &  6469 & reVMdu \\
    47541 & kA &  5961 & ixi \\
    43075 & ko &  5768 & kAni \\
    32438 & hEM &  5686 & uMtuMxi \\
    29251 & para &  5524 & koVnni \\
    28067 & BI &  5293 & wana \\
    26270 & nahIM &  5264 & axi \\
    24800 & ki &  4516 & lo \\
    23185 & eka &  4483 & e \\
    22507 & hI &  4367 & ane \\ \hline
\end{tabular}
\end{center}\end{table}
Size of corpora (total number of words contained in each corpus
if all the words in all the texts in the corpus are added), is
given below.

\begin{center}
Total number of words (size of corpus)\\[0.2em]
\begin{tabular}{cc}
  Hindi Corpus       &        Telugu Corpus\\
  2,863,157 	&	     3,097,668 \\
\end{tabular}
\end{center}

A frequency list can be used to prepare children's books graded
on vocabulary, to develop morphological analyzers, to chose words
for inclusion in a dictionary, etc. (Complete lists of high-frequency
words in seven languages can be obtained by anonymous ftp, as described
in Conclusions at the end of this paper.)

If we look at the total number of distinct words in Hindi and Telugu
corpora, we find:

\begin{center}
Total number of distinct words\\[0.2em]
\begin{tabular}{cc}
  Hindi corpus       &        Telugu corpus\\
   120,227	 &     	     544,641\\
\end{tabular}
\end{center}

\vskip-4pt
Therefore, for roughly the same size of corpora (3 million words),
the number of distinct words in Hindi is only about 25 percent of
that in Telugu. Similarly, if we see the count of number of words
with low frequency, say, with the frequency of 1, we find:

\begin{center}
Total number of words with frequency 1\\[0.2em]
\begin{tabular}{cc}
  Hindi corpus       &        Telugu corpus\\
    61,510       &            372,463 \\
\end{tabular}
\end{center}

A comparision between the above information between Hindi and Telugu
yields results that we would expect. In Telugu, the vibhakti of a
noun or a verb is written together with the root in a single word;
whereas in Hindi, this is mostly not so. (Vibhakti for a noun means 
its case ending or post-position as the case may be; 
similarly for a verb, vibhakti stands for its tense aspect and 
modality. For a discussion on the concept of vibhakti, see Bharati et al (1995).)
In Hindi, the vibhakti is separated and 
placed as a post-position marker after the word. Therefore, the
number of distinct words in Telugu is much larger because root plus
vibhakti in a single word gives rise to many different distinct words.

\section{Coverage}

Using the word-frequency list, the coverage of a corpus can be found.
For example, the first word 'ke' in Hindi occurs 103216 times in a
corpus of 2,863,157 words. Thus, it covers 3.32 percent of the corpus.
Similarly, the first two words (ke and hE) occur 190675 (103216 +
87459) times and cover 6.14 percent of the corpus, as shown in
Table 2 (word frequency with cumulative coverage).

\begin{table}[hbt]
\begin{center}
    Table 2: Word-frequency with cumulative coverage \\
            for Hindi and Telugu\\[0.2em]
\begin{tabular}{rrlrrl} \hline
\multicolumn{3}{c}{ Hindi  Corpus } &\multicolumn{3}{c}{Telugu Corpus} \\
Coverage &  Frequency & Word & Coverage & Frequency & Word \\ \hline 

     3.32\% & 103216 & ke	&	1.07\% & 29548 & I\\
     6.14\% &  87459 & hE	&	1.73\% & 18306 & A\\
     8.75\% &  81128 & meM	&	2.33\% & 16429 & oVka\\
    11.03\% &  70685 & kI	&	2.68\% &  9886 & kUdA\\
    12.72\% &  52507 & se	&	3.02\% &  9289 & ani\\
    14.39\% &  51630 & Ora	&	3.25\% &  6469 & reVMdu\\
    15.92\% &  47541 & kA	&	3.47\% &  5961 & ixi\\
    17.30\% &  43075 & ko	&	3.68\% &  5768 & kAni\\
    18.35\% &  32438 & hEM	&	3.88\% &  5686 & uMtuMxi\\
    19.29\% &  29251 & para	&	4.08\% &  5524 & koVnni\\
    20.20\% &  28067 & BI	&	4.28\% &  5293 & wana\\
    21.04\% &  26270 & nahIM	&	4.47\% &  5264 & axi\\
    21.84\% &  24800 & ki	&	4.63\% &  4516 & lo\\
    22.59\% &  23185 & eka	&	4.79\% &  4483 & e\\
    23.31\% &  22507 & hI	&	4.95\% &  4367 & ane\\
\hline
\end{tabular}
\end{center}
\end{table}

\vskip-5pt
If we wish to find out the number of distinct words, that give
a specified coverage of the corpus, then we can look at the
frequency list and find out the requisite number. For example, Table 3
gives the coverage by the words in the frequency list of Hindi and Telugu.

\begin{table}[hbt]
\begin{center}
    Table 3: Coverage of Hindi and Telugu Corpora \\ [0.2em]
    Number of distinct words for different coverage\\
    levels (10\%, 20\%, etc.) of the respective corpora\\
\begin{tabular}{crr}
\hline
 Coverage & Hindi &  Telugu\\
\hline
    10\% :- &       4 &       63\\
    20\% :- &      11 &      295\\
    30\% :- &      29 &      953\\
    40\% :- &      83 &     2488\\
    50\% :- &     210 &     5858\\
    60\% :- &     519 &    13728\\
    70\% :- &    1238 &    33335\\
    80\% :- &    3089 &    89391\\
    90\% :- &   10165 &   268137\\
\hline
\end{tabular}
\end{center}
\end{table}

Thus, a 10\% coverage of the Hindi corpus is given by a mere 4 words, and 
this 
number remains unchanged even if the corpus size is fictitiously reduced by 
taking a subsample of the corpus (See Table 5 ).
Using the same method, we can find the coverage table for each of the
seven corpora and present it as a comparative chart in Table 4.

\begin{table}[hbt]
\begin{center}
    Table 4: Coverage table for seven languages\\[0.2em]

    Number of distinct words for different coverage\\
    levels (10\%, 20\%, etc.) for the respective corpora\\
\begin{tabular}{crrrrrrr}
\hline
Coverage & Punjabi & Hindi & Bengali & Oriya & Marathi & Kannada & Telugu\\
\hline
10\% :- &     5 &     4 &    17 &    20 &    17 &       34 &       63\\
20\% :- &    15 &    11 &    61 &    78 &    71 &      187 &      295\\
30\% :- &    32 &    29 &   173 &   199 &   218 &      586 &      953\\
40\% :- &    88 &    83 &   419 &   444 &   556 &     1520 &     2488\\
50\% :- &   232 &   210 &   910 &   894 &  1251 &     3571 &     5858\\
60\% :- &   594 &   519 &  1878 &  1792 &  2838 &     8126 &    13728\\
70\% :- &  1533 &  1238 &  3950 &  3714 &  6657 &    19350 &    33335\\
80\% :- &  3958 &  3089 &  9150 &  8653 & 17047 &    51674 &    89391\\
90\% :- & 12451 & 10165 & 27755 & 26962 & 54067 &   162893 &   268137\\
\hline
\end{tabular}
\end{center}
\end{table}

Assuming that each corpus is representative of the language, the
general trends shown in Table 4 are no different from what we would 
expect regarding the
different languages. As discussed while comparing Hindi and Telugu,
those languages in which their vibhakties are written as case endings
(merged with the word), it requires much larger number of distinct
words to achieve the same coverage. Telugu and Kannada are at one
extreme of the scale, and Hindi and Punjabi are at the other. Bangla,
Oriya and Marathi are somewhere in between. Marathi has apparently more
forms than Bengali and Oriya, and Telugu has more forms than Kannada.
This also gives an indication of the difficulty of building a
morphological analyzer for these languages.

If the size of the corpus is increased, it seems unlikely to have an effect 
on the number of distinct words needed to cover upto a certain percentage of 
corpus. Beyond that percentage,  the number of words needed is likely to 
increase rapidly as the corpus size increases. This, in our opinion, reflects
the nature and the size of the corpus rather than property of the language.
We explored this by 
considering the coverage tables for subsamples of smaller sizes from a given 
corpus and carrying out an analysis.  

For instance, for the Hindi Corpus, we started with a randomly selected 
subsample of size approximately 0.8 milliion from the corpus.  Subsequently 
about 0.1 million words were added to it at each stage by randomly choosing 
the texts from the rest of the corpus, till the total corpus was exhausted. 
At each stage, the coverage table was constructed by merging the 
frequency list of the newly added texts list to the existing frequency list
at the earlier stage.  This procedure gave us coverage tables for 23 
samples for sizes of 0.8, 0.9 till 3.0 million words.
 
A part of the table is presented in Table 5  for Hindi corpus.
	
\begin{table}[hbt]
\begin{center}
Table 5: Number of Distinct Words for Fixed Precentage Coverage \\[0.2em]
\begin{tabular}{ccccccccr} \\ \hline
Corpus size & \multicolumn{8}{c}{P : Percentage Coverage of the Corpus} \\
(in millions) & 10\% & 50\% & 70\% & 80\% & 90\% & 95\% & 99\% & 100\% \\ \hline
0.8 &    4 &  198 & 1161 & 2809 & 8366 & 18032 & 42970 &  50906 \\
1.2 &    4 &  200 & 1173 & 2882 & 8927 & 20327 & 53831 &  65717 \\
1.6 &    4 &  198 & 1173 & 2909 & 9227 & 21481 & 62668 &  78494 \\
2.0 &    4 &  198 & 1176 & 2936 & 9460 & 22594 & 70256 &  90053 \\
2.4 &    4 &  197 & 1177 & 2950 & 9662 & 23402 & 77377 & 101136 \\
2.8 &    4 &  197 & 1176 & 2961 & 9775 & 23953 & 82803 & 110496 \\
3.0 &    4 &  197 & 1177 & 2966 & 9829 & 24198 & 85664 & 115333 \\ \hline
\end{tabular}
\end{center}
\end{table}

It is clear from the table that even if the corpus size is increased, 
the number of distinct words needed to cover upto 70\% of corpus remains
almost the same.  For values of P from 80 to 100, we investigated the patterns
of increase in the number of distinct words for P\% coverage with increase
in corpus size. A nonlinear regression equation was fitted to the data and 
the details of the statistical analysis are presented in 
Appendix at the end. Figure 1 presents the observed values (denoted by 
'.' ) of number of distinct words for 80\% coverage against corpus size 
(in millions of words) and the fitted regression curve (solid line -) for P 
equal to 80\% for Hindi corpus.
 
\begin{figure}[hbt]
\begin{center}
Fig 1 : The Regression Curve. 
\end{center}
\vskip6cm
\end{figure}

The analysis can be used to predict the upper bound on the number of 
distinct words for P percent coverage for various corpus sizes by
taking larger and larger corpus sizes. As the
corpus size goes to infinity, the number of distinct words for P\% coverage 
gives us a measure for the entire language. 
For instance, for Hindi, the approximate 95\% prediction upper bound on the 
number of distinct words for 80\% coverage of the total language 
is about 3258. This information 
is useful for various purposes such as estimating the difficulty of builing 
a morphological analyzer with 80\% coverage.

The nature of increase in the number of distinct words for 80\% coverage 
as the corpus 
size increases can be seen from Figure 2, where solid line
denotes the prediction curve and '*' denote the observed values.

\begin{figure}[hbt]
\begin{center}
Fig 2 : The Prediction Curve.
\end{center}
\vskip6cm
\end{figure}

 
In this particular example the fact that prediction is achieved with 
satisfactory accuracy is because  after a certain value,
the larger corpus size does not change the number of distinct words by a
larger margin, which can also be seen from the nature of the data points  
in Figure 1. If the number of distinct words increase with a higher rate 
as the corpus size increases, which typically happens for larger values of
P,  the statistical analysis does not give a satisfactory regression fit. 

For instance, for Hindi corpus for P=99\%, the regression fit is highly 
unsatisfactory, giving very wide margins of prediction bounds,
which can be intuitively seen from the plot of corpus size against the
number of ditinct words for 99\% coverage presented in Figure 3. 

\begin{figure}[hbt]
\begin{center}
Fig. 3 : Plot of number of distinct words for P=99\% against corpus size.
\end{center}
\vskip6cm
\end{figure}

One might expect that for modern languages like Hindi, the total number of 
distinct words is large but finite. Therefore, as the corpus size goes to 
infinity, for P=100\% there must be a definite upper limit on the number
of distinct words. However, the fact that this number could not be predicted
even for P=99\% reflects upon the small size of the corpus and the consequent
lack of corresponding observations. In future, when larger corpora are 
available, it may be possible to predict this number. 

\section{Common Words Across Languages}

The next question that is explored is: what is the overlap of
vocabulary across languages, in other words, how many words are common 
across pairs of languages. The following experiment was conducted:
from the list of most frequently used words in a corpus, first N words
are chosen which would provide 60\% coverage. Let us call these the word
lists. For example, the following were the values of N for the word
lists for the different languages (taken from Table 4):

\begin{center}
\begin{tabular}{crrrrrrr}
Coverage & Punjabi & Hindi & Bengali & Oriya & Marathi & Kannada & Telugu\\
60\% :- &   594 &   519 &  1878 &   1792 &   2838 &   8126 &    13728\\
\end{tabular}
\end{center}

Now, given a text in a language, each of the words in the text are
looked up in all the word lists, and a count is kept for words found
in each of the word lists. At the end, the percentage of words found for
each of the word lists is calculated. This experiment was conducted
by taking six randomly chosen texts in different languages (from
the corpora) and taking the average across the six samples for each
language. The figures for common words are given in  Table 6.

\begin{table}[hbt]
\begin{center}
            Table 6: Common words across languages \\[0.2em]

  Percentages of words in a random text in a language which \\
  were found to occur in word lists of length N for different \\
  languages. (N was chosen for each language to give a coverage of
  60\%.) \\
\begin{tabular}{rrrrrrrl}
\hline
Punjabi & Hindi & Bengali & Oriya & Marathi & Kannada & Telugu &  Input-text-in \\
\hline
    56 &     21 &     12 &      7 &     17 &     10 &     12 &      /punjabi\\
    29 &     58 &     22 &     18 &     30 &     29 &     32 &      /hindi\\
     6 &      8 &     58 &     24 &     10 &     11 &     13 &      /bengali\\
     4 &      7 &     23 &     55 &      9 &     15 &     14 &      /oriya\\
     4 &      9 &      7 &      7 &     54 &     14 &     13 &      /marathi\\
     4 &      6 &      4 &      5 &      9 &     58 &     16 &      /kannada\\
     3 &      4 &      2 &      4 &      5 &      9 &     54 &      /telugu\\
\hline
\end{tabular}
\end{center}
\end{table}

It is  noticed, as expected, that Punjabi and Hindi have many common
words. Assuming that the corpus is representative of the language,
if the experiment is conducted with a randomly chosen Punjabi text,
56\% words in the text occur in the Punjabi word list and 21\% words
from the same text occur in the Hindi word list. If a Hindi text is
chosen the corresponding figures are: 29\% and 58\%. Similarly, Bengali
and Oriya turn out to be close, and so on for Telugu and Kannada.
Marathi shows a slightly greater similarity with Kannada and Telugu than
the other languages. 

The limitations of these experiments must be clearly understood.

(i) Words are defined to be a sequence of characters separated by
spaces or punctuation marks (definition given at the beginning of
the paper). Equality of root is not considered. Thus, if a root is
the same (or sounds similar) in two languages, but takes a slightly
different ending in the two languages, the machine will not find
any similarity between the corresponding words, with the above
method. (Thus, these results cannot directly be used for a study
of common roots or cognates.) To do the same experiment on roots
using the machine, morphological analyzer for each of the languages, 
would be needed.

(ii) If a word is found to occur in two word lists, it is assumed
to be common to the two. If the meanings are different in the two,
for example, the spellings by chance happen to be the same,
this assumption will not be correct.

The above results are used for building a language detector. Given
a text whose language is not specified, the same experiment can
be conducted and the name of the language can be found for which
highest percentage of words are found. Such a program is available
as free software. The implemented version, called {\it find\_code\_lang},
identifies the coding scheme used (out of a given set of schemes
including the ISCII schemes), as well as the language (out of the
above 7 languages) for a given text. 

\section{Conclusions}

This paper presents some preliminary results obtained using the
computer and corpora of some Indian languages. Two kinds of data are
extracted: (1) Number of words needed to obtain designated level of
coverage of a language, and (2) percentage of common words across
Indian languages. The general trends are as expected. Some quantitative
results are presented. The latter has been used to build a language
detector for detecting the name of the language of a given unknown text
(when the language is not specified). The results of this paper can be
easily checked for other Indian languages, when corpora of other Indian
languages are available.

If morphological analyzers for each of the languages are available,
they can be used to produce data not just on identity of words but
also similarity among roots across languages. Development of such
analyzer is a basic need which must be undertaken in a spirit of
"community work" by the linguists and computer scientists working
together. There is also a great need to build bilingual electronic
dictionaires. Morphological analyzers for some of the languages above
are being developed as part of anusaaraka systems (as part of a joint
project between IIT Kanpur and University of Hyderabad funded by TDIL,
DOE, Government of India).

All the data and systems developed as part of the anusaaraka project
are available as free software. Frequency lists for each of the
languages are also available for free distribution. They may
be down loaded for free under GPL license by anonymous ftp from:
anu.uohyd.ernet.in
\vskip5pt

\centerline{\bf Acknowledgement}
\vskip3pt

Thanks are due to the following organizations for providing the
corpora: CIIL Mysore for Telugu and Kannada corpora, WRLC Pune
for Marathi corpus, IIT Delhi for
Hindi and Punjabi corpora, and IIALS Bhubaneshwar for Bengali and
Oriya corpora. These corpora were developed under funding from TDIL,
Dept of Electronics (DOE), Government of India. CIIL, Mysore is now
the repository of all the corpora, on behalf of DOE.

The major support for the work reported in this paper has come
from TDIL, DOE. RS is now being supported by Satyam School at
I.I.I.T. Hyderabad.

Frequency lists for the corpora were extracted using programs written
by Mrs Amba P Kulkarni. Thanks are also due to Mrs. Nisha Sangal and
Mr Vikas Bansal for doing most of the programming needed for the extraction 
of the above data.

\section{ Appendix }

 We present the details of the statistical analysis of the behaviour of 
number of distinct words to cover P\% of the corpus as the corpus size 
increases.
 
 Let $ t $ be the size of corpus and $Y(t,P)$ be the
number of distinct words which cover P\% of the corpus of size $ t $. 
The data collected were on the number of distinct words which  cover P\%
of the corpora of various sizes, obtained on adding 0.1 million words to the 
existing corpus at each stage as described in Section 3.

We denote by  $t_1,\cdots, t_n$ the successive corpus  sizes (in millions) and
the corresponding observations on number of distinct words by 
$Y(t_1,P), \cdots, 
Y(t_n, P)$ for every value of P. We assume $$ Y(t,P) =
 g(t,P,\underline{\theta}) + \varepsilon_t \eqno $$
where $g(t,P,\underline{\theta})$ is a function of $(t, P)$ and unknown
parameters $\underline{\theta}$ and $\varepsilon_t$ is the random 
error. Empirical evidence indicates that $g(t,P,\underline{\theta})$ 
can be free from $t$ for 
small values of $P$, such as  P=10 for which we have 
$g(t,10, \underline{\theta}) = 4$ (See Table 5).

For a chosen value of P, the scatter plot of $t_i$ against $Y(t_i,P)$ reveals 
the nature of change in $Y(t,P)$ as $t$ increases and can be seen from Figure 1
for P=80. Suppose a particular value of $P$ is chosen. For notational 
simplicity,
 we drop P from the expressions henceforth. The behaviour of the rate of
change in $Y(t)$ with respect to $t$ can be seen from the plot of empirical 
growth rate $X_{t_i} = Y(t_{i+1}) - Y(t_i), ~~ i=1,\cdots,n-1$ against the 
corpus size $t_i$ which helps in determining the form of 
$g(t,P,\underline{\theta})$ in (1). 

%Figure 4  shows such a plot for P=80 for Hindi corpus.

%\begin{figure}[hbt]
%\begin{center}
%Fig 4: Plot of consecutive differences of number of words against corpus size.
%\end{center}
%\vskip7cm
%\end{figure}


Based on the empirical evidence, we consider the stochastic differential 
equation 
$$dY(t) = g(t,\underline{\theta})dt + d\varepsilon_t$$
which can alternatively be written as $X(t) = f(t,\underline{\beta}) + W_t$
where $X(t) = dY(t)$ and $W_t = d\varepsilon_t$ and $\underline{\beta}$ 
represents the unknown parameters.
 
A nonlinear regression of $\{X(t)\}$ on $t$ was fitted for every fixed value 
of P using Splus package. For Hindi as well as Pujabi corpus, for instance, 
it was found that the regression equation 
$$X(t) = \beta_1{(1+t)^{\beta_2}} + W_t ; ~~~~~~\beta_1 >0,~~\beta_2<0$$
gives a good fit. That the fit is satisfactory for the specified value of $P$
was confirmed by investigating
the coefficient of determination and residual analysis. For example, in case 
of Hindi corpus, for P=80\% the coefficient of determination is  83.48\% 
and decreases for larger values of P indicating that the fit is less 
satisfactory. For P=90\% it is 68.83\% and reduces
to 53\% for P=95%. 

The residual plots showed  random scatters and  this empirical evidence led us
 to assume that the original error process $\{\varepsilon_t\}$ is 
nonstationary and autocorrelated with 
stationary and independent increments. Under this assumption, the stochastic
differential equation in (2) can be solved using Ito integrals and we get the
solution process $\{Y(t)\}$ to be 
$$ Y(t) = Y(t_0) + \int_{t_0}^{t}f(u,\underline{\beta}) du +
 \int_{t_0}^{t}dW_u $$.

% and for $t_1<t_2<\cdots<t_n$ we get 
%$$ Y(t_{i+1}) = Y(t_i) + \int_{t_i}^{t_{i+1}}f(u,\underline{\beta}) du + 
%\int_{t_i}^{t_{i+1}}dW_u $$

The solution process was used to predict the values of $Y(t)$, with a bias
correction for values of $t$ larger than the available corpus size and one 
such prediction curve for P=80\% is presented in Figure 2. 
We also computed the
prediction intervals and as anticipated, the prediction intervals were found 
to be wider for larger values of P. Based on the analysis, we conclude that
upto P=85\%, the data give reliable information for Hindi language. Upto 
$P=90\%$ the results can be used with some caution. For P beyond 
95\%, no reliable conclusions could be drawn. 

\section{ References}

{\bf[Bharati et al, 1995]}
Bharati, Akshar, Vineet Chaitanya, and Rajeev Sangal, Natural
Language Processing: A Paninian Perspective, Prentice-Hall of India,
New Delhi, 1995.

{\bf[Bharati et al, 1998]}
Bharati, Akshar, Vineet Chaitanya, Amba P Kulkarni, Rajeev Sangal,
and G Uma Maheshwar Rao, Anusaaraka: Overcoming the Language Barrier
in India, In Anuvad: Approaches to Translation, Rukmini Bhaya Nair
(editor), Katha, New Delhi, 1998 (Forthcoming).

{\bf[Ratkowsky, 1983]}
Ratkowsky, David A., Nonlinear Regression Modelling : A Unified Practical 
Approach, Marcel Dekker, New York, 1983.
\end{document}
