The origin of the ninja is based on the spies and assassins that have existed throughout Japanese history. The title ninja has sometimes been attributed to the semi-legendary 4th century prince Yamato Takeru. In the Kojiki, the young Yamato Takeru disguised himself as a charming maiden, and assassinated two chiefs of the Kumaso people. However, these records take place at a very early stage of Japanese history, and is unlikely to be connected to the shinobi of later accounts.
The first recorded use of espionage was under the employment of Prince Shoutoku in the 6th century. Such tactics were considered unsavory even in early times, when, according to the 10th century Shoumonki, the boy spy Koharumaru was killed for spying against the insurgent Taira no Masakado. Later, the 14th century war chronicle Taiheiki contained many references to shinobi, and credited the destruction of a castle by fire to an unnamed but "highly skilled shinobi".
However, it was not until the 15th century that spies were specially trained for their purpose. It was around this time that the word shinobi appeared to define and clearly identify ninjas as a secretive group of agents. Evidence for this can be seen in historical documents, which began to refer to stealthy soldiers as shinobi during the Sengoku period. Later manuals regarding espionage are often grounded in Chinese military strategy, quoting works such as The Art of War (Sunzi Bingfa), by Sun Tzu.
The ninja emerged as mercenaries in the 15th century, where they were recruited as spies, raiders, arsonists and even terrorists. Amongst the samurai, a sense of ritual and decorum was observed, where one was expected to fight or duel openly. Combined with the unrest of the Sengoku era, these factors created a demand for men willing to commit deeds considered not respectable for conventional warriors. By the Sengoku period, the shinobi had several roles, including spy (kanchou), scout (teisatsu), surprise attacker (kisho), and agitator (koran). The ninja families were organized into larger guilds, each with their own territories. A system of rank existed. A jounin ("upper man") was the highest rank, representing the group and hiring out mercenaries. This is followed by the chuunin ("middle man"), assistants to the jounin. At the bottom was the genin ("lower man"), field agents drawn from the lower class and assigned to carry out actual missions.
In the early 18th century, shogun Tokugawa Yoshimune founded the oniwaban, an intelligence agency and secret service. Members of this office, the oniwabanshuu ("garden keeper"), were agents involved in collecting information on daimyos and government officials. The secretive nature of the oniwaban — along with the earlier tradition of using Iga and Kōga clan members as palace guards — have lead some sources to define the oniwabanshuu as "ninjas". This portrayal is also common in later novels and jidaigeki. However, there is no written link between the earlier shinobi and the later oniwabanshuu.