In 1863 the Bakufu recruited ronin to guard Shogun Iemochi on a visit to Kyoto to meet with the Emperor Komei. This visit was a precedent breaking event—not since the third Shogun of the Tokugawa Bakufu, Tokugawa Iemitsu, had a reigning shogun gone to Kyoto. This was a difficult time for a Japan as the country was violently struggling to find consensus on how to deal with the threat sparked by the arrival of American and then European naval squadrons demanding that Japan open up or face military action. Tokugawa Iemochi, as head of the military government, was being summoned to confer on how to enact the recent imperial edict calling for the expulsion of foreigners to be backed up by the use of force.
Matsudaira Katamori, daimyo of Aizu han was given the newly created post of Protector of Kyoto and charged with the responsibility of curtailing the lawlessness that had gripped Kyoto as well as guaranteeing the Shogun’s safety during his stay in Kyoto. As the anarchy being wreaked in Kyoto in the name of Sonnô Jôi (revere the Emperor; expel the foreigners) by pro-imperial ronin, the Bakufu felt that the best way to fight ronin was with other ronin. The actual plan for the Rôshigumi is credited to Matsudaira Chikaranosuke, chief Kenjutsu instructor at the Shogunate’s military academy Kôbusho. This new corps of pro-Bakufu ronin was named the Roshigumi and Kiyokawa Hachirô of Shonai han, was given the responsibility of recruiting members. Kiyokawa Hachirô was chosen to recruit ronin for the newly created Roshigumi while Yamaoka Tesshu, Kiyokawa's longtime friend, provided support and additional leadership. In reality, Kiyokawa harbored anti-Tokugawa sentiments and was a vehement supporter of the principles of sonnô jôi and proceeded to recruit like-minded ronin to fill the Roshigumi’s ranks. Additionally, Kiyokawa secretly planned to turn the Roshigumi into an instrument of sonno joi upon arrival in Kyoto, abandoning the mission of protecting Iemochi. With this plan in mind, Kiyokawa marched out of Edo with a force of 250 men on February 8, 1863, as the vanguard of Shogun Iemochi’s procession to Kyoto.
Not long after arriving in Kyoto, Kiyokawa made his intentions regarding his sonno joi plans for the Rôshigumi clear. This did not come as a surprise to some senior Bakufu officials, who long regarded Kiyokawa as a dangerous subversive. Anxious to get Kiyokawa and his men out of the explosive situation in Kyoto, orders were arranged telling Kiyokawa to bring the Rôshigumi back to Edo to partake in the military preparations for expelling the foreigners. However, thirteen of the Roshigumi refused to return to Edo and petitioned Matsudaira Katamori to stay in Kyoto in order to complete their original mission of protecting the Shogun.
Thirteen ex-Roshigumi were bolstered by the arrival of five new recruits and hence named the Mibu Rôshigumi, (also known as the Mibugumi), after the village of Mibu on the outskirts of Kyoto where they were headquartered. Matsudaira Katamori, after careful evaluation of the political scene in Kyoto, felt it was needed to change the scope of the Mibu Roshigumi's mission from one of protecting the Shogun to one of patrolling the streets of Kyoto and restoring order in the name of the Bakufu. To reflect the change in mission, on August 18, 1863, the Mibugumi was re-named the Shinsengumi— "Newly Selected Corps".
Out of the remnants of the Roshigumi who refused to return to Edo, the Shinsengumi was born. Matsudaira Katamori named three commanders: Kondô Isami, leader of the eight-man Shieikan faction; Serizawa Kamo leader of the five-man Mito faction; and Niimi Nishiki, another member of the Mito faction who was only a nominal commander and did not wield any true power. Kondô and Serizawa were fierce rivals and the tension between the two was reaching a boiling point. Kondô and his right hand man, Vice-Commander Hijikata Toshizô, began to plot the destruction of the Mito faction. Their first break came in early September 1863, when Niimi was found guilty of extorting money for use at the geisha houses and was forced to submit Seppuku. Serizawa's violent and unruly behavior gave Kondô the opportunity he needed to finish the job and seize sole power. As it was felt that Serizawa was soiling the group's reputation, Matsudaira Katamori, ordered the assassination of Serizawa and his closest cohorts. On September 16 or 18 (there isn’t clear agreement on the exact date), Hijikata, the gifted swordsman Okita Sôji and two other members loyal to Kondô assassinated Serizawa and Hirayama Gorô. A third assassination target, Hirama Jusuke, escaped. With the Mito faction broken, Kondô and Hijikata had absolute control over the Shinsengumi.
Ikedaya Incident Anti-Bakufu elements were banished from Kyôto after the Political change on August 18 (1863), however rather than be driven from Kyoto, they went underground. The Shinsengumi investigators Yamazaki Susumu and Shimada Kai located a merchant by the name of Masuya Kiemon who was actually the Ômi Gôshi Furutaka Shuntaro, who was cooperating with Chôshû. On the morning of 1864/6/5, Takeda Kanryusai and several Shinsengumi members raided and captured Masuya, and found arms and armor and letters from Chôshû samurai. Furutaka was taken to the Shinsengumi headquarters for questioning. He was tortured and revealed that the Sonnô Jôi Rôshi planned to wait for a windy night to set fire to the city of Kyoto and in the confusion kidnap the Emperor and take him to Chôshû. The story was immediately reported to the Military Commissioner of Kyoto, and a raid on the Ikedaya took place the same day.
Around 4:00PM, on 1864/6/5, Matsudaira Katamori ordered a search for the Rôshi. Kondô Isami the commander of the Shinsengumi waited for reinforcements from Katamori but they never came, so he gathered 30 Shinsengumi members at the Gion Kaisho (PM8:00) and separated into two groups and started searching.(PM10:00)
Kondo took Okita Sôji, Nagakura Shinpachi, Tôdô Heisuke, Kondô Shûhei, Takeda Kanryûsai, Tani Mantarô. Asano Kaoru, Andô Sôtarô, Okuzawa Eisuke, Nitta Kakuzaemon. Hijikata Toshizô took the rest.
At Ikedaya Inn, there were about 20 extreme Sonnô Jôi Rôshi discussing what to do next. Their plan was to destroy the Shinsengumi headquater and bring Chôshû troops into Kyoto.
Kondô left Ando, Okuzawa, Nitta at the backyard of Ikedaya, Takeda, Asano, Tani at the front door and entered with Okita, Nagakura and Tôdô.
When Sôbe the master of Ikedaya warned to the Rôshi on upstairs, Kondô and Okita ran up to the stairs and told "Anybody who resist will be cut off.". Some of the confused Roshi jumped onto the backyard, Okita killed one but he fell down unconscious and left Ikedaya with Tôdô who was injured forehead.
Only Kondô and Nagakura were inside of Ikedaya, fighting against many Rôshi. Then Hijikata turned up with reinforcements and the Ikedaya was surrounded by Aizu and Kuwana troops. The battle ended in 2 hours.
Okuzawa was killed, Ando and Nitta were heavily injured and died later.
According to Kondô's letter, 7 Sonnô Jôi Rôshi were killed, 23 arrested. And the Ikedaya Sôbe's family and relatives were arrested.
The Shinsengumi members who participated in the raid received a bonus from Bakufu. This incident later caused the Kinmon Rebellion.
January 1868, the Boshin War was started with the Battle of Toba-Fushimi. The Shinsengumi lost almost 100 members, including Inoue Genzaburô and Yamazaki Susumu. The Shinsengumi and other Bakufu troops fled to Edo with the battleship Fujisan Maru. After this, the Shinsengumi was reorganized into the Koyo Chinbutai. The Koyo Chinbutai intended to take over Kofu castle, but lost the Battle of Koshu-Katsunuma. When they returned to Edo, Nagakura Shinpachi and Harada Sanosuke left the Koyo Chinbutai and formed the Seikyotai. Kondo was captured in Nagareyama three weeks later and executed at Itabashi. Harada died from injuries received during the Battle of Ueno. Two weeks later, Okita Sôji died of what is thought to be Tuberculosis. Hijikata Toshizô was wounded at the Battle of Utsunomiya castle, and Saitô Hajime then became the commander of the Aizu Shinsengumi during the Battle of Aizu, and decided to remain with the Aizu samurai when Hijikata decided to go to Hakodate. In May 1869 Hijikata was shot and killed at the Battle of Hakodate and the last Shinsengumi commander, Soma Kazue, surrendered. He was later exiled to Niijima island.