Stylometry and Ḥadīth Corpus:
In a recent discussion "Refuting Orientalists" (@qadissiyah636 - RO) argued that the Reliability of the Sunnī Ḥadīth corpus can be established through a stylometry. This is problematic on multiple levels. In this thread, we will show what...
In a recent discussion "Refuting Orientalists" (@qadissiyah636 - RO) argued that the Reliability of the Sunnī Ḥadīth corpus can be established through a stylometry. This is problematic on multiple levels. In this thread, we will show what...
exactly is the problem with this claim.
First of all, it is important to discuss the first study cited by RO. This study claims to apply stylometry on Qur'an and Ḥadīth to conclude that Quran and Ḥadīth have two different authors. It is clear that Qur'an and Ḥadīth have...
First of all, it is important to discuss the first study cited by RO. This study claims to apply stylometry on Qur'an and Ḥadīth to conclude that Quran and Ḥadīth have two different authors. It is clear that Qur'an and Ḥadīth have...
two different producers and styles. The real question is on Ḥadīth having a single author.
The problem here is that to establish single authorship, there needs to be a strong analysis of the Hadith corpus itself that takes into account all the details. Mere similarity in the...
The problem here is that to establish single authorship, there needs to be a strong analysis of the Hadith corpus itself that takes into account all the details. Mere similarity in the...
cluster does not entail single authorship. There ought to be a model that has been independently tested to recognize single authorship (i.e., after feeding it data, it can recognize multiple textual blocks of the same text as well).
This needs to be done for multiple texts of the same genre. A positive result on such a model would entail much higher confidence in the conclusions. Such a model is not available in this study.
The model presented in this study has shown no quantification or analysis about why it should be trusted on the claim of single authorship. RO claimed that this is "peer-reviewed". As any good researcher would tell you, just because a paper is peer-reviewed does not mean that...
supposed conclusions are necessarily true, unless they can be reliably recreated.
Even if this study is taken at the very best, even then it merely takes 4 blocks from Ṣaḥīḥ Bukhārī for analysis. That is an extremely limited dataset to demonstrate any general conclusion.
Even if this study is taken at the very best, even then it merely takes 4 blocks from Ṣaḥīḥ Bukhārī for analysis. That is an extremely limited dataset to demonstrate any general conclusion.
Actual Stylometric analysis to establish single authorship is much more difficult. For example, a team of researchers at Cornell have shown that there are great similarities between various different Greek authors who wrote in the same genre. 
This shows that even very good ML-based models are unable to distinguish accurately within different authors between the same genre and style. If this is the case for more established linguistic models, one can only expect Arabic models to act worse, not better.
Storey & Mimno (2020) have shown how two authors, even centuries apart, can have high similarity in their style and content. Such similarity can entail "single authorship" in any unsophisticated model that aims to establish single authorship. 
Therefore, for us to trust a model, it *must* show that it accurately predicts single authorship for texts that are single author and multiple authorship for texts that are known to be multiple authors. A good test would be distinguishing between Kalām texts that use...
similar vocabulary but different styles in a nuanced way. If a model has accurately matched known data for particular genres, then it becomes much more reliable. This is not the case for the model being cited by RO. Rather, we have no evidence whatsoever that this model can...
even accurately determine authorship in the first place. It was mainly used to establish different clusters of Quran and Ḥadīth, not to prove that Ḥadīth itself has merely one author.
The study cited by RO themselves show that there is variation between the Ḥadīth clusters. As it can be seen in figure 2, there is a substantial variation within the cluster. This is the case when not all "Ṣaḥīḥ" corpus is used. 
In addition to this, we know for a fact that at the level of syntactic construction, the Ḥadīth corpus does not, at the very least, represent peculiar Prophetic syntactic constructions. That is to say, even if the best case is granted, and it is assumed that the meaningful...
content of Ḥadīth indeed originates from the Prophet, the peculiar constructions still are very unlikely to be from the Prophet. This is due to the simple reason that majority of narrations have gone through multiple layers of redaction, where each redactor slightly changes.
This is easily testable by collating asānīd (chains) of any Ḥadīth. A natural variation would be observed. This would render word frequency analysis, or analyses for some peculiar constructions impotent, as we know for a fact that these features of a Ḥadīth do not originate...
from the Prophet, even if the meaningful content does.
RO further argued that there are other studies and models that have successfully differentiated within Arabic texts. First of all, this is irrelevant to the original claim. Even if there is a Stylometric model that is...
RO further argued that there are other studies and models that have successfully differentiated within Arabic texts. First of all, this is irrelevant to the original claim. Even if there is a Stylometric model that is...
100% accurate, it would not establish single authorship for Ḥadīth unless that particular model itself is applied to the data. Accuracy of model X is not going to make the conclusions of model Y true. Of course, that is simple logic. The two models are independent.
Yet, it is important to analyze the claim of such an accurate model.
Even the studies RO cited show extreme limitation in their analysis. First of all, they did not run their model multiple times on multiple texts, as would be expected from a reliable model.
Even the studies RO cited show extreme limitation in their analysis. First of all, they did not run their model multiple times on multiple texts, as would be expected from a reliable model.
Furthermore, they already agree that they are limited. Thirdly, they used multiple blocks from the same text, which makes matching much easier. I addition to this, they were not testing for "single authorship" but rather authorship matching/prediction, where the assumption is...
that one of the available options would match. These models do not show the sophistication whatsoever to complexly differentiate between textual so reliably that they can establish single authorship. Anyone trained in statistics or computational philological disciplines can...
confirm this.
RO also presented a study that there is a model that can differentiate between Ṣaḥīḥ Ḥadīth and Muwḍūʿ Aḥādith. This is indeed true. What RO does not realize is that successful predication and classification does not entail single authorship.
RO also presented a study that there is a model that can differentiate between Ṣaḥīḥ Ḥadīth and Muwḍūʿ Aḥādith. This is indeed true. What RO does not realize is that successful predication and classification does not entail single authorship.
In this study, there were multiple differences between the very content of Mawdūʿ Aḥādith that would make them detectable. This is not an argument for single authorship of Ṣaḥīḥ Ḥadīth. This is an argument for internal differentiation between the Ṣaḥīḥ corpus and the... 
the so-called Muwdūʿ corpus. This can be fully explainable by different intellectual origins for their development. One can see the high usage of the name of ʿAlī in the "fabricated" corpus. This is evident because a Ḥadīth is usually deemed fabricated due to Shīʿī narrators.
Overall, RO has failed to demonstrate strongly how Sunnī Ḥadīth corpus has a single author. What can be said is that Ṣaḥīḥ Sunnī Ḥadīth corpus has common features, but that is the case for corpus that RO would deem "unreliable" as well, such as Mawdu or Shī'ī corpus.
When best ML models cannot accurately establish single authorship for a widely studied canon, what makes RO think that Arabic stylometry, a deeply understudied and underdeveloped field can establish single authorship for a corpus that essentially lacks idiosyncratic features?
RO fails to realize that presenting 9 extremely limited studied on Arabic stylometry will not prove anything. Especially, they would not change the accuracy of the *actual* model used for analysis.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
