gzxq

Title: acoustic word embeddings for end-to-end speech synthesis (end-to-end speech synthesis based on acoustic word embedding)

Journal: Applied Sciences

Authors: Feiyu Shen, Chenpeng Du and Kai Yu,

Publication date: 27 September 2021

DOI：10.3390/app11199010

Wechat link:

https://mp.weixin.qq.com/s? __biz=MzI1MzEzNjgxMQ==&mid=2650042375&idx=2&sn=

151c5c62ae0d5583c474b10c284d5428&chksm=f1d9cac3c6ae43d556bf64fb78ad7179

03d668684fdcafaee7dd0ffd1d686fc17a573d1c6c20&token=497878244&lang=zh_CN#rd

Journal link:

https://www.mdpi.com/journal/applsci

Introduction of communication author

Professor Yu Kai

Shanghai Jiao Tong University

The main research direction is the research and industrialization of interactive artificial intelligence, speech and natural language processing and machine learning.

Speech synthesis is one of the key issues to realize human-computer interaction. Yu Kai’s team from the Cross-media Speech Intelligence Laboratory of Shanghai Jiaotong University recentlyApplied SciencesIn this paper, an end-to-end speech synthesis method based on acoustic word embedding is studied.

introduction

In recent years, the end-to-end text-to-speech synthesis (TTS) model based on sequence-to-sequence generation architecture has achieved great success in generating natural speech. In order to avoid frame-by-frame decreasing, unnatural TTS models, such as FastSpeech and FastSpeech2, are proposed to improve the speed of speech generation. However, vocabulary recognition is very important for TTS systems to generate highly natural speech, but most end-to-end TTS systems only use phonemes as input markers, ignoring the information of which vocabulary phonemes come from.

Previous studies used preset language words to embed phoneme sequences as the input of TTS system. However, because language information is not directly related to how words are pronounced, the embedding of these words hardly affects the improvement of TTS quality. In this paper, a word acoustic embedding method combined with TTS system is proposed. Experiments on LJSpeech data set show that the acoustic embedding of words significantly improves the likelihood of prosodic prediction of sonic level on training set and verification set. The subjective evaluation of the naturalness of synthesized audio shows that the system with acoustic word embedding is obviously superior to the pure TTS system and other TTS systems with preset word embedding.

design highlights

1. Propose an end-to-end TTS system based on Acoustic Word Embedding (AWE) joint training to help improve rhythm and naturalness;

2. Using the prosody modeling method based on GMM to objectively measure the prosody of synthesized audio;

3. Explore the best structure of word encoder and the best threshold of word frequency screening;

4. Subjective evaluation after the test shows that AWE is more natural than pre-trained word embedding.

End-to-end speech synthesis method based on acoustic word embedding

Part1: end-to-end speech synthesis

In this paper, FastSpeech2 is selected as the acoustic model, but prosody modeling is not explicitly considered, which makes it difficult to objectively evaluate the prosody prediction performance of TTS system without subjective listening test. Based on this, a phoneme-level prosodic prediction module is introduced into the model, which can predict the prosodic embedding distribution of each phoneme autoregressively. Compared with the standard Fastspeech2 system, it can not only improve naturalness, but also allow the logarithmic likelihood of prosodic embedding to easily and objectively evaluate the prosodic prediction performance at phoneme level, as shown in Figure 1.

Figure 1. End-to-end speech synthesis based on GMM prosody modeling.

Part2: Acoustic Word Embedding

Nowadays, most popular TTS systems use phonemes as sound input markers, ignoring the information of which vocabulary phonemes come from. However, vocabulary recognition is very important for TTS system to generate highly natural speech. In this paper, acoustic word embedding is proposed to synthesize natural speech, and a word encoder and a word phoneme aligner are introduced into the traditional TTS system. The architecture is shown in Figure 2.

Figure 2. Model architecture with acoustic word embedding.

Experimental setup and results

1. Experimental setup

The TTS model in this paper is based on Fastspeech2 (GMM prosodic modeling). The number of Gaussian components in GMM is set to 20, and Adam optimizer and Noam learning rate scheduling strategy are used for TTS training. The researchers used 320mel-spectrogram as acoustic features, with a frame shift of 12.5ms and a frame length of 50 ms.. MelGAN is used as vocoder to reconstruct waveform.

2. Word encoder architecture

The performance of three common word encoder structures is compared.

(1) None: the baseline of the word encoder is not used;

(2) BLSTM；: a layer of 512-dimensional bidirectional LSTM;

(3) Transformer: 6-layer 512-dimensional transformer module;

(4) Conv+Transformer: one layer of 1D-CNN with a core size of 3, followed by six layers of 512-dimensional Transformer module.

Table 1. Parameter number and reasoning speed of various encoder architectures (sec/frame).

3. Word frequency threshold

Table 2. Vocabulary and OOV under different word frequency thresholds.

Fig. 3. Loglikelihood curve of phoneme prosody with different word frequency thresholds.

conclusion

In this paper, an innovative method of directly training acoustic embedding of words in TTS system is proposed. The phoneme sequence and the word sequence pass through two encoders respectively, which are used as the input of TTS system together, and then the two output hidden states are spliced together for phoneme-level prosodic prediction. Experiments on LJSpeech data set show that the superposition structure of convolution and Transformer is the best for word encoder. In addition, the selection of word frequency threshold should be cautious, too large or too small threshold will lead to performance degradation. Finally, this paper compares the proposed system with several works that do not use the baseline of lexical information and use pre-trained words to embed. The subjective listening test shows that the system proposed in this paper is superior to all other systems in naturalness.

Applied SciencesPeriodical introduction

Editor-in-Chief: Takayoshi Kobayashi, The University of Electro-Communications, Japan.

The subject of the journal covers all aspects of applied physics, applied chemistry, engineering, environmental and earth sciences and applied biology.

　　CCTV News:On December 13th, the Taiwan Province Affairs Office of the State Council held a regular press conference to answer reporters’ questions on recent cross-strait hot issues.

　　A reporter asked: New Year’s Day, Spring Festival, winter vacation and other festivals and holidays are approaching, and many Taiwanese businessmen and Taiwanese students in the mainland are very concerned about the air tickets to Taiwan. What are the arrangements between the two sides of the strait for the winter vacation and the Spring Festival this year?

　　Zhu Fenglian, spokesperson of the Taiwan Affairs Office of the State Council, responded: The Spring Festival is the most important traditional festival of the Chinese nation, and the vast number of Taiwanese students, Taiwanese businessmen and Taiwan compatriots hope to return home smoothly for the holidays and family reunion. We feel the same way about their wishes. Relevant mainland authorities will actively make preparations for cross-strait Spring Festival overtime flights in line with the purpose of serving compatriots on both sides of the strait and facilitating the return of Taiwan compatriots. But to solve this problem, we also need the cooperation of Taiwan Province. It is hoped that Taiwan Province will listen more to the reflection of Taiwanese businessmen and Taiwan compatriots, come up with attitudes and practices that meet people’s expectations, and don’t deliberately set up obstacles to prevent the vast number of Taiwanese businessmen and Taiwan compatriots from returning home to reunite with their families.

　　　　　　 CCTV News:Yesterday (5th), the national railways adjusted the train diagram. After this adjustment, Spring Festival travel rush this year is a little different from previous years. This first "different", summed up in four words, is an increase of "one-stop direct". According to the adjusted train diagram, it is possible to have a one-stop direct train from Shenyang to Beijing. From 0: 00 yesterday (5th), Shenyang North Station started a "one-stop direct" high-speed train to Beijing South Station for the first time, with a running time of 3 hours and 45 minutes, which is also the shortest running time between Shenyang and Beijing. There is a direct stop, and there is a direct connection between the two places. This year, the high-speed train from Fujian will go through Guangxi and Sichuan for the first time in Spring Festival travel rush.

Image source network

　　　　　　 [Direct connection between the two places] Fujian opened the first direct high-speed train in Sichuan, Guangxi.

　　　　　　 After the implementation of the new train operation map for the national railways on January 5, Fujian started the first through high-speed trains from Fuzhou to Nanning East and from Xiamen North to Chengdu East. This means that this year in Spring Festival travel rush, the high-speed train from Fujian will go directly to Guangxi and Sichuan provinces for the first time. Among them, the whole running time from Fuzhou to Nanning is about 10 hours and 30 minutes, saving 2 hours compared with before; The whole running time from Xiamen to Chengdu is about 13 hours and 40 minutes, saving nearly 4 hours compared with before.

　　　　　　 It is understood that the high-speed train from Fuzhou to Nanning East will pass through Jiangxi and Hunan; The high-speed train from Xiamen to Chengdu will pass through Guangxi, Hunan, Guizhou, Chongqing and other places, all of which are routes with more tourists from Spring Festival travel rush in previous years. It is predicted that the peak of returning home before the holiday will appear from January 28th to February 3rd in Spring Festival travel rush this year.

　　　　　　 [Interconnection of Three Places] From scratch, Tianjin Jizhou entered the era of high-speed rail.

　　　　　　 In the past, it was inconvenient to travel from Beijing or Hebei to Jizhou, Tianjin, and the journey of reversing many times made people very tired. However, after the national railway adjusted the train operation map yesterday (5th), Jizhou District of Tianjin welcomed the train stop for the first time, and it only took 42 minutes to get from Beijing to Jizhou at the earliest. In addition, in order to facilitate the travel of foreign tourists to Jizhou, a bus line connecting Jizhou City and Jizhou Station is set up in the square of Jizhou Station. Through the seamless connection between buses and trains, tourists can reach tourist attractions and mountain village farmhouses more conveniently.

　　　　　　 At present, there are five EMU trains stopping at Jizhou Station in Tianjin, connecting Jizhou with Beijing and Qinhuangdao in Hebei, making the rail transit network between Beijing, Tianjin and Hebei more dense and convenient.

　　　　　　 [Severe punishment for "bully"] 452 people were arrested for "bully" and other disorderly behaviors.

　　　　　　 There is another difference in Spring Festival travel rush this year, that is, strengthening the rectification of the "hegemony" behavior. Recently, the "bully seat" and "bully shop" have become a hot issue of social concern. In order to create a safe and orderly travel environment for the vast number of tourists, the railway police authorities across the country have recently launched special actions to focus on rectifying outstanding public security problems that disrupt the order of stations and cars, such as "bully seat", "blocking doors" and "forcibly entering the station". Since the launch of the special campaign of centralized rectification, the railway police have investigated and dealt with 2,856 illegal and disorderly behaviors, and 452 people have been administratively detained by the railway police according to law for disturbing the order of stations and cars, such as "dominating seats".

　　　　　　 On December 1, 2018, on the G170 train, a drunken woman forcibly occupied other people’s seats and insulted the staff and other passengers. Subsequently, the Beijing Railway Public Security Bureau immediately contacted and coordinated with the Shanghai Railway Public Security Bureau to jointly handle it properly. Nanjing railway police took the woman out of the car at Nanjing South Railway Station, punished her by administrative detention for 7 days according to law, and informed the railway credit information system.

　　　　　　 On December 8, 2018, on the Z25 train from Wuchang to Shanghai, another "Bapu" man appeared. After the man arrived at the station, Ba refused to get off the bus in the sleeper compartment and refused to make up the ticket, and his attitude was bad. The man’s behavior seriously disturbed the normal order and was administratively detained by the Wuhan Railway Police according to law.

　　　　　　 On December 14, 2018, on the K7426 train, a woman held a ticket for car No.4, but forcibly occupied three seats in car No.3, causing three passengers to be unable to sit. After being persuaded and educated by the train conductor and the police, the woman still refused to give up her seat. Subsequently, the Jilin Railway Public Security Bureau took him away from his seat by police. The bully woman was punished by administrative detention for 5 days.

　　On the afternoon of September 27th, the State Council Press Office held a routine briefing on the State Council policy, introducing the situation of supporting the stable development of foreign trade and answering reporters’ questions.

　　Wang Shouwen, international trade negotiator and vice minister of the Ministry of Commerce, said that from January to August, China’s foreign trade import and export reached 27.3 trillion yuan, a record high. However, foreign trade is facing some new uncertainties, and the growth of external demand is slowing down, which is the biggest uncertainty facing foreign trade at present. This is reflected in the slowdown in the economic growth of major economies such as the United States and Europe, and the decline in the growth rate of global trade in goods.

　　In order to cope with this background and promote the steady growth of foreign trade, the CPC Central Committee and the State Council made timely arrangements to create a good environment for foreign trade enterprises and create good conditions for the goal of maintaining stability and improving quality of foreign trade.

　　From January to August, China’s mechanical and electrical products accounted for 56.5% of foreign trade exports, while automobile exports increased by 57.6% and solar cell exports increased by 92.6%. It is worth noting that in August, the export growth of China to other RCEP member countries reached 23%, which was 11.2 percentage points higher than the overall foreign trade export growth rate in August. Free trade agreements, including RCEP, came into effect on January 1st this year, which played a very important role in promoting the development of China’s foreign trade.

　　Generally speaking, despite the background of slowing or even weakening external demand, we are confident that foreign trade will achieve positive growth in the second half of the year and make positive contributions to the recovery of the national economy through the growth of foreign trade.

　　Beijing, Jan 23rd (Leng Haoyang) In 2018, Spring Festival travel rush’s "ticket grabbing war" is still hot. However, for the passengers who haven’t grabbed the train tickets, they can find another way to re-plan their Spring Festival reunion. At this time, while paying attention to the "missing" of the remaining tickets, such passengers can also skillfully use air-rail combined transport, continuous transfer, network hitchhiking, reverse family visits and other measures to "curve" reunion.

　　Data Map: On January 4th, Taiyuan Railway Station, Shanxi Province, passengers bought train tickets at the manual ticketing window. China News Service reporter Wei Liang photo

　　Always pay attention to the remaining tickets and prepare for "missing"

　　For passengers who haven’t grabbed their tickets, according to the introduction of the railway department, we can pay attention to several time points when there are more refunds.

　　First, 30 minutes after the ticket is sold, if a passenger fails to pay in time after successfully grabbing the ticket, these ticket sources will return to the ticket warehouse after 30 minutes;

　　Second, on the 15th day before driving, according to the relevant regulations on railway refund, refund fees are not charged for those who refund tickets more than 15 days before driving, and passengers with uncertain trips often choose this time node to apply for refund.

　　The third is one to two days before driving. At this time, passengers who occupy tickets can basically determine the itinerary, especially from 22: 00 to 23: 00, which is the peak time for passengers to refund their tickets;

　　Fourth, before driving, the system will release some of the remaining tickets, and passengers who have not grabbed the tickets can "catch up" at this time.

　　In addition, the reporter noticed that some trains temporarily stopped selling tickets due to map adjustment, and some temporary passenger trains were added by the railway department. Passengers who did not grab tickets can pay more attention to the ticketing time of such trains in order to grab tickets at the first time.

　　Clever use of air-rail combined transport can "curve home"

　　In Spring Festival travel rush, long-distance train tickets are generally "one ticket is hard to find". In this regard, passengers can re-plan the route, air-rail combined transport, and curve home.

　　Nowadays, many airlines and railway departments have launched air-rail combined transport, flying to cities near their destinations first, and then transferring to high-speed trains to go home, or taking a high-speed train first, and then transferring to planes to fly to their destinations.

　　As for the cost, compared with the railway and Spring Festival travel rush, there will be some situations in air transportation where the capacity of some routes is relatively sufficient and the discount is large. Choosing air-rail combined transport is not only easy to buy tickets but also affordable.

　　Taking Beijing to Ningbo as an example, the reporter’s inquiry showed that on February 14th (the 29th day of the twelfth lunar month), not only was it hard to get a train ticket, but it also cost more than 1,000 yuan to fly. If you fly to Shanghai first on February 14th, the cheapest air ticket is only 400 yuan, and then transfer to a high-speed train with a price of 100 yuan, the whole journey will only take you around 600 yuan, which not only reduces the difficulty of purchasing tickets, but also makes it more cost-effective.

　　Try "connecting and transferring" and buy more stops.

　　The recommendation of the "continuous transfer" scheme introduced by the railway department last year can also reduce the difficulty for passengers to purchase tickets. When passengers can’t buy through tickets, they can choose the "continuous transfer" function, and the ticketing system will show passengers the remaining tickets of some trains that transfer once on the way. If passengers choose to buy, they can complete the payment of tickets for two trips at one time.

　　The reporter inquired about the train ticket from Beijing to Lanzhou on February 14 at the 12306 client, and found that there were only 8 trains a day on this line, and there were no tickets left. The reporter also inquired about the tickets from Beijing to Xi ‘an on the same day, and found that among the 32 trains, some high-speed trains still have tickets available for purchase.

　　The reporter also inquired about the tickets from Xi ‘an to Lanzhou, and found that there were 76 trains on February 14th, and there were still many tickets left. If you transfer home in this way, it will make it easier for passengers to buy tickets.

　　In addition, passengers can also try to buy more stops. According to the principle of "ensuring long distance" of the railway department, passengers can try to buy a few stops backwards when purchasing tickets, and get off early, or try to buy tickets from the previous stops to increase the chances of buying their favorite tickets.

　　You can choose a long-distance "ride" and pay attention to safety.

　　In recent years, the rise of long-distance "hitchhiking" has also greatly facilitated passengers’ way home during the Spring Festival.

　　Zhang Peng works in Beijing and lives in Jiyuan, Henan. Zhang Peng said that although the distance is not far, there is no direct high-speed train, and there are only five trains a day, which takes about 10 hours. If you choose to take the high-speed train to Zhengzhou and then transfer, on the one hand, it is not easy to buy tickets, on the other hand, it is more troublesome.

　　Zhang Peng told the reporter of Zhongxin. com that he has been a frequent visitor to the long-distance ride. He used the ride to return home during the 11th holiday last year, and he still plans to take the ride this Spring Festival. Although the fare of more than 300 yuan is twice as high as 178 yuan for a hard sleeper, it is convenient and worry-free.

　　On the client side of a network car, just select the "cross-city" option in the "Ride" interface, then enter the departure time, departure place and destination to get a car, and you can also see the estimated price. From Fuchengmen, Beijing to Jiyuan Bus Station, the estimated cost is 341.8 yuan.

　　The safety of the network car has also been concerned. In this regard, the police reminded that when taking a car online, it is best to write down the license plate number before getting on the bus. It is best not to carpool with people you don’t know and share your position with relatives and friends at any time. In case of danger, you should keep calm and call the police in time.

　　"reverse visiting relatives" to realize another kind of reunion

　　Wang Bo, a native of Harbin, has worked in Beijing for three years. In the previous two years, he was a member of Spring Festival travel rush’s "ticket grabbing army", but he plans to invite his parents to Beijing for the New Year this year.

　　Wang Bo told the reporter of Zhongxin. com that going home every year is very "heart-tired". Tickets are too difficult to grab, and it is inevitable to visit relatives and friends when going home. It has become a luxury to take advantage of the Spring Festival holiday to have a good rest.

　　"My father has always wanted to come to the National Centre for the Performing Arts to watch a performance, just taking advantage of the Spring Festival to take his parents around Beijing. During the Spring Festival, there are fewer Beijingers, and a family of three can have a good year in Beijing quietly." Wang Bo said.

　　In this regard, the reporter inquired about the train ticket from Beijing to Harbin on February 14 and found that "one ticket is hard to find", and even the economy class of the plane is not much left. However, if there is a "reverse reunion", the ticket source from Harbin to Beijing is sufficient, and the minimum flight cost is only 200 yuan. (End)

　　Press conference site

　　CCTV News:On October 13th, the State Council held a press conference on import and export in the first three quarters. In response to a reporter’s question, Li Kuiwen, spokesman of the General Administration of Customs and director of the Statistics and Analysis Department, introduced that the Customs resolutely implemented the decision-making arrangements of the CPC Central Committee and the State Council, concentrated on "six stables" and "six guarantees", made overall plans to prevent and control the epidemic situation at ports and promote the steady growth of foreign trade, further proposed 56 measures to stabilize foreign trade and foreign investment, and continuously improved the level of customs clearance facilitation.

　　First, reform and innovation promote trade facilitation.We will comprehensively promote facilitation measures such as "two-step declaration", "declare in advance", "green channel" and exemption from on-site inspection to improve customs clearance efficiency. In September, the overall customs clearance time for imports and exports in China was 35.02 hours and 1.86 hours respectively, which was 64.04% and 84.87% lower than that in 2017, both of which have decreased for five consecutive months.

　　The second is to find ways to reduce the cost of enterprises.By reducing and exempting the import delay payment, late payment fee and the interest of deferred tax on domestic sales of processing trade goods, the company has reduced and saved nearly 200 million yuan. Especially among the import and export enterprises that use tariff guarantee insurance, SMEs account for about 70%.

　　The third is to accelerate the pilot to promote the development of new formats.Comprehensively promote the regulatory measures for the return of cross-border e-commerce export commodities, and carry out pilot projects for cross-border e-commerce enterprises to export to enterprises. Encourage the central and western regions, the northeast and other places to carry out market procurement trade pilot. In the first three quarters, the import and export through the customs cross-border e-commerce management platform was 187.39 billion yuan, up 52.8%, and the market procurement export was 509.86 billion yuan, up 35.5%.

　　The fourth is to take multiple measures to promote the creation of a new highland for opening up.Vigorously support the construction of Hainan Free Trade Port and Yangshan Comprehensive Bonded Zone, intensify the innovation and replication of customs supervision system in the Pilot Free Trade Zone, and promote the full implementation of global maintenance and remanufacturing business in the comprehensive bonded zone. In the first three quarters, the import and export of the national comprehensive bonded zone was 2.34 trillion yuan, an increase of 15.1%, 14.4 percentage points higher than the growth rate of foreign trade.

　　The fifth is to fully guarantee the import of key products.We will continue to expand the scope of importing countries and commodities, newly approve 5,965 foreign registered enterprises, and increase the import security of important agricultural products and foods such as meat, grain, aquatic products and fruits. In the first three quarters, meat and grain were imported 162.65 billion yuan and 253.36 billion yuan respectively, up by 87.3% and 20.7% respectively.

　　In the next step, the national customs will deepen reform and innovation, do a good job in the work of "six stabilities" and "six guarantees", fully promote the implementation of 56 measures to further stabilize foreign trade and foreign investment, promote the stable development and quality improvement of foreign trade, and make positive contributions to promoting the formation of a new development pattern.

　　According to The New York Times’s report on July 13th, SAG-AFTRA announced that it had failed to reach a new agreement with AMPTP, which represents the interests of film and television production companies, and decided to strike.

　　Earlier, the American Writers’ Union (WGA) began a strike on May 2. SAG-AFTRA will soon join their camp, which is also after 1960, when actors and screenwriters went on strike again. During the strike, the actors’ union will stop all script, film and TV production under AMPTP.

　　Last month, members of the Actors’ Union voted overwhelmingly by 98%, authorizing the union management to go on strike if no agreement can be reached in the negotiations, and at the same time authorizing the union leaders to "stick it out" in the negotiations with the management.

　　In the latest statement, the Actors’ Union said: "SAG-AFTRA is eager to reach an agreement that fully meets the needs of performers, and the producer’s response to the most important proposal of the Actors’ Union is to insult and disrespect our contribution to this industry. These companies refuse to make meaningful contacts on some topics, while others completely hinder us. "

　　The union also said that after more than four weeks of negotiations, AMPTP is still unwilling to provide a fair deal on key issues that are crucial to the members of the Actors Union. "

　　AMPTP represents Amazon, Apple, Disney, Comcast’s NBC Universal, Netflix, Paramount Universal, Sony and Warner Bros. Discovery and other major film companies and streaming media companies.

　　It is reported that the demands of the actors’ union include "economic fairness, late dividends, standardizing the use of AI, and alleviating the burden of self-recorded audition fees in the whole industry".

　　The core issue is salary and the use of artificial intelligence.

　　The actors’ union believes that the remuneration of actors has "seriously shrunk" in recent years. Under the old system, if a TV series is a great success, actors can expect that there will be a large amount of checks left in their bank accounts in the next few years. Actors believe that in the era of streaming media, cakes become smaller and checks become smaller.

　　"Fundamentally, we are interested in ensuring that our members share the success of the projects they create." Duncan Crabtree-Ireland, the chief negotiator of the trade union, pointed out.

　　Actors are also very concerned about artificial intelligence and how to use this technology to replicate their previous performances without paying or consulting them.

　　The strike of the actors’ union means that almost all American TV dramas and movies being filmed will stop shooting. During the strike, hundreds of thousands of actors who joined the trade union were not allowed to shoot any films and TV works, and could not attend any press conference or film premiere.

　　21st century business herald reporter Ji Yuanyuan reports from Shanghai Recently, Yixinhe Pharmaceutical Technology Co., Ltd. (hereinafter referred to as "Yixinhe Medicine") developed and produced, and (1) Shakuba Quvalsartan Sodium Tablets (brand name: Yixintan) which was exclusively commercialized was approved by National Medical Products Administration (NMPA). Yixintan is the first generic drug of shakubatravalsartan sodium tablets approved for marketing in the global regulatory market by breaking through the original research patent through innovative crystal forms. The approved indications of this product are for treating essential hypertension and adult patients with chronic heart failure (NYHA II-IV grade, LVEF≤40%) with reduced ejection fraction, so as to reduce the risk of cardiovascular death and hospitalization for heart failure.

　　Immediately afterwards, Unacon also announced that its subsidiary company Shiyao Ouyi developed Shakubatrivalsartan sodium tablets (50mg, 100mg, 200mg) which have obtained the drug registration approval issued by National Medical Products Administration, and regarded them as the adult patients with chronic heart failure (NYHA II-IV grade, LVEF≤40%) with reduced ejection fraction, which reduced the risk of cardiovascular death and hospitalization for heart failure.

　　With the approval of domestic shakubatravalsartan sodium tablets, Novartis, the original drug manufacturer, also released information at the first time, saying that the marketing license for the generic drug of Novartis’ original research product shakubatravalsartan sodium tablets was approved, and Novartis declared that the pharmaceutical active ingredient of the original research product Noxintuo was protected by patents ZL200680001733.0 and ZL201210191052.2 in China.

　　According to the relevant provisions of Article 11 of China’s Patent Law, any enterprise or individual shall not manufacture, use, promise to sell (including applying for online registration, participating in bidding, etc.), sell or import products falling within the scope of the above patent protection before the expiration of the above patent period, otherwise it will constitute patent infringement. To this end, Novartis emphasizes that it will reserve the right to claim its patent right against infringing imitations from relevant judicial and/or administrative organs.

　　Regarding the patent issue, according to the public information, in April 2017, the patent of Shakubatroxobin Sodium Tablets registered by Novartis was declared invalid in China. On July 2, 2021, the crystal patent was also partially invalidated in China. Yixin and Medicine also emphasized in the official disclosure information that in 2022, the domestic sales of Shakuba Quvalsartan sodium tablets totaled nearly 3 billion yuan, and its China patent will expire in November 2026. Yixintan is a first-line drug for the treatment of heart failure and hypertension with independent intellectual property rights, which can legally make the majority of patients with heart failure and hypertension in China use equivalent and affordable innovative drugs three years in advance.

　　It can also be seen that the domestic Shakubatroxobin Valsartan sodium tablets have been approved one after another, which will bring certain market challenges to the original drug.

　　More than 10 pharmaceutical companies compete for the market

　　Shakubatroxobin and valsartan sodium tablets are a new type of eutectic drug composed of enkephalinase inhibitors Shakubatroxobin and angiotensin receptor blocker valsartan sodium tablets in a molar ratio of 1:1. Eutectic structure can make the absorption and elimination rates of shakubarb and valsartan sodium tablets similar, and ensure the synchronous consistency of their efficacy. In addition, compared with independent preparations, eutectic drugs have the advantages of constant composition ratio, good storage stability, and can significantly improve drug solubility and oral bioavailability.

　　As a representative drug of ARNi, Shakubatroxobin sodium tablets have been unanimously recommended by domestic and foreign guidelines in recent years. In 2021, the American Heart Association proposed that ARNi drugs should be given priority to patients with HErEF, and it is recommended to use them to the maximum tolerated dose or target dose (Class I). In 2018, China Heart Failure and Treatment Guidelines also recommended to replace ACEI/ARB with ARNi to further reduce the incidence and mortality of heart failure (Class I).

　　Sakubatravalsartan sodium was originally researched by Novartis Entresto, and was first approved by the FDA in July 2015. It is the world’s first breakthrough and innovative drug in the field of heart failure treatment, and was approved to enter the domestic market for the first time in July 2017. It was included in the national medical insurance category B catalogue through negotiation in 2019, successfully renewed in 2021, and added indications for essential hypertension. In 2021, the global sales of the original research sand kubatroxobin sodium reached US$ 3.548 billion. In 2022, its global sales scale reached US$ 4.644 billion, up 37% year-on-year. In the first half of 2023, the drug also grew into the best-selling drug of Novartis, with sales reaching US$ 2.915 billion, up 31% year-on-year, and it is expected to exceed US$ 6 billion in the whole year.

　　In the China market, the original yanshakubatravalsartan sodium is also a heavy variety. According to the data of the Minenet, the sales growth rate of this product in China’s three major terminals and six major markets has reached triple digits in recent years. In 2021, it exceeded 2 billion yuan for the first time, up 127.62% year-on-year, with full market potential; In 2022, the sales of Shakubatroxobin Sodium Tablets in China’s three major terminals and six major markets exceeded 2 billion yuan. Among them, the hospital market (public hospital terminal+public primary medical terminal) is the main sales channel, with a year-on-year increase of 6.54%.

　　In this regard, some brokerage analysts told 21st century business herald that the innovative product layout must be aimed at chronic diseases, major diseases, diseases that pose a major threat to people’s health, such as cardiovascular diseases, tumors and major infectious diseases. At the same time, the core of making medicine is clinical value. With the rapid development of life science, many interdisciplinary fields are very active, and the large-scale treatment methods focusing on large populations are worthy of attention.

　　"The successive approval of domestic shakubatravalsartan sodium tablets is a great benefit for cardiovascular patients, although the layout of many domestic pharmaceutical companies is already in the clinical stage, except for the single-minded and pharmaceutical/Fosun Pharma, Shiyan Ouyi, including Sichuan Kelun, Chengdu Beite, Qilu Pharmaceutical, Zhengda Tianqing, etc. are also accelerating the layout. This is due to the accelerated layout of many domestic pharmaceutical companies, and the prices of related products are expected to be further lowered, eventually benefiting more patients. " The above analyst said.

　　According to the data of Minenet, at present, there are three companies that have production approval documents for Shakuba Quvalsartan Sodium Tablets. Among them, Unacon Ouyi Pharmaceutical Co., Ltd. and Nanjing Yixin and Medical Science and Technology Co., Ltd. were all approved on the same day with four types of imitation, which are regarded as over-evaluation, and tied for the first imitation and the first over-evaluation. In addition, 17 enterprises, including Shandong New Times Pharmaceutical, Nanjing Zhengda Tianqing Pharmaceutical and Qilu Pharmaceutical (Hainan), all reported their production under the new classification.

　　An unavoidable patent dispute

　　The competitive anxiety between Chinese and foreign pharmaceutical companies in Shakuba Quvalsartan sodium tablets is also considered to be an important reason for patent confrontation. At the same time, in recent years, patent disputes between original drugs and generic drugs have occurred from time to time. 21st century business herald reporter found through the search of Wisdom Bud database that as of the latest, there are more than 700 drug patents with litigation results in China, and more than 99% of them are invention patents. It can be seen that the handling of drug patent disputes needs urgent attention.

　　Yixinhe Medicine also emphasized in the official information disclosure that Yixintan is a first-line drug for the treatment of heart failure and hypertension with independent intellectual property rights. Compared with the reference crystal form, when the relative humidity changes from 20% to 60%, it absorbs less water and is more stable, and can be sealed and stored below 30℃. This stable crystal form is more suitable for the vast China market.

　　When talking about patents, some pharmaceutical executives also told 21st century business herald that the research and development of original innovative drugs has always been a "three high" industry with high threshold, high risk and high return. According to the latest literature, it takes an average of 12.5 years for a new drug to go on the market, which costs nearly $2.6 billion, and the success rate is less than 10%. Every new drug project is the most important decision of an enterprise, and the project process needs a lot of data research, especially the research on patents, documents and regulations takes a lot of time.

　　"Therefore, how to find a balance between ensuring the innovation enthusiasm of the original drug research enterprises and reducing the burden on patients has been hotly debated. The FDA’s definition of generic drugs is very strict, and the requirements must be consistent with the dosage form, specifications and administration route of the copied products. If it is inconsistent, it will be declared as a new drug. " The above-mentioned pharmaceutical company executives said that for the long-term public interest, the state needs to protect the enthusiasm of the original drug research and development enterprises. At the same time, in order to limit the original drug research and development enterprises, it is necessary to encourage the imitation research and development of the original drug from the level of institutional norms and ensure the interests of generic drug enterprises.

　　In order to better balance the interests of the original research drug and generic drug companies with the system, the new Patent Law, which came into effect on June 1, 2021, established the drug patent linking system in China, aiming at linking the two legal procedures, so as to clarify the interest relationship between the original research drug manufacturers and generic drug manufacturers and avoid the problem of patent punishment when the generic drug research and development is completed.

　　As for how enterprises can avoid patent risks, Hou Xuchao, the founding partner of CIC Consulting, also told 21st century business herald that from the perspective of investment and financing, pharmaceutical companies have five key parameters when choosing product pipelines, including the success rate of products under research, patient base, product penetration rate, product pricing and patent life. With the improvement of pharmaceutical companies’ R&D strength, patent life needs more and more attention.

　　"For the patent situation of unlisted drugs, investors need to hire professional IP lawyers to conduct detailed investigations. Because patent conflicts will directly make it difficult for candidate drugs to be marketed, and even directly affect the product value measured by all the above parameters, it is an essential step for patent investigation. " Hou xuchao said.

　　There is a huge crowd in the chronic disease market behind Shakuba Quvalsartan sodium tablets. In terms of hypertension alone, China, as a big country with hypertension, has 245 million adult hypertensive patients, which means that Shakuba Quvalsartan sodium tablets have a broad market space and are an important direction for the future development of drugs. I believe many domestic and foreign manufacturers will make layouts and list related products one after another.

　　However, it should also be noted that after the appearance of the first imitation drug, the clinical data, approval time and product price of the follow-up drugs and enterprises will all be factors affecting the competition pattern, especially the clinical effectiveness and safety, which will be the key to decide who can go further in this track.

　　CCTV News:Today, the China Meteorological Bureau held a consultation meeting with various departments on the weather during the Spring Festival holiday. According to the forecast, during the holiday, the temperature in most parts of the north was low, while the temperature in the south was high and rainy. From New Year’s Eve to the second day, the weather in most parts of the country is generally conducive to travel. From the third day, affected by strong cold air, there will be a large range of rain, snow and strong winds in the central and eastern regions.

　　According to the forecast, from the 4th night today, due to the influence of a medium-intensity cold air, there will be 4 to 6 northerly winds from north to south in the central part of northwest China, central and eastern Inner Mongolia, most of the northeast, Jianghuai, Jiangnan and eastern Guizhou. The temperature will drop by 4 ~ 8℃, and the local temperature will drop above 12℃. Everyone should pay attention to wind protection and warmth when traveling.

　　From New Year’s Eve to the first day, there is no obvious cooling and snowfall in most parts of the country, which is conducive to travel. Around the second day of junior high school, a strong cold air will affect northern Xinjiang, bringing strong winds, strong cooling and snowfall. From the third day to the fifth day, this cold air will continue to move eastward and affect the central and eastern regions, bringing northerly winds of 4 ~ 6 grades, and the temperature will also drop by 6 ~ 10 C. Among them, the local cooling in central and southern Hunan and southeastern Guizhou will exceed 14 C. At the same time, there is snow or sleet weather in the eastern part of northwest China, north China, south-central part of northeast China, Huanghuai, Jianghuai and Jianghan. From the fifth day to the sixth day, there are northerly winds with 6-8 grades and gusts of 9-10 grades in Bohai Sea, Yellow Sea, East China Sea, Taiwan Province Strait, northern South China Sea and Beibu Gulf.

　　During the Spring Festival holiday, there will be rainy weather in the south. It is expected that there will be more light rain or showers in Jiangnan, northern South China and eastern Guizhou, and the number of rainy days will be 4-6 days.

　　In addition, it should be noted that there will be heavy fog in Qiongzhou Strait on New Year’s Eve and the early morning of the first day, and the local visibility will be less than 500 meters. On the second day, there are heavy fog in Sichuan Basin, Henan, Shandong, Jiangsu, Anhui, Hubei and other places, and there are strong fog in some areas, so it is necessary to guard against the adverse effects of low visibility on travel. (CCTV reporter Ren Meimei)

Piano robot Special Olympics is playing and singing.

The painting of Microsoft robot "Xiao Bing"

"Xiao Bing" solo exhibition held in Central Academy of Fine Arts

Participants in the "to be no.1" game at the World Artificial Intelligence Conference.

The audience participated in the live interaction of Li Yuansu’s AI work "Nanjing West Road Style Migration"

　　Yangcheng Evening News reporter Jing He

　　"Humans can’t be controlled by artificial intelligence, because humans can’t create smarter creatures than humans." Ma Yun said.

　　"I disagree with you very much. We can create something smarter than human beings, but not necessarily people." Musk said.

　　On August 29th, at the opening ceremony of the 2019 World Artificial Intelligence Conference in Shanghai, Ma Yun and Tesla CEO elon musk had a tit-for-tat "quarrel".

　　Regardless of whether artificial intelligence is smarter than human beings, it is an undeniable fact that AI is infiltrating into every corner of human life, even including the field of artistic creation that was once regarded as exclusive to human beings. If the cooperation between artists and AI only creates a new means of expression, but AI itself learns to create art, which opens the door to a new world.

　　What does AI’s participation mean for creative and imaginative humanistic artistic creation? How should we define "art" today? Is it possible for everyone to become an artist in the future? Artists who are at the forefront of the trend have given us their own answers.

　　1 Smart conference live experience: somatosensory remote control and AR "crossing" are human-computer interaction.

　　Walk into the IBM exhibition area of the 2019 World Artificial Intelligence Conference and raise your hands above your head to make a gesture of love. With the countdown to 54321 over, you will successfully activate an astronaut named CIMON to "land on the moon". On the screen, the astronaut will follow some of your body movements to "swing" his body — — He is representing you "landing on the moon". You can also download photos of yourself "landing on the moon" and post them on social platforms. This CIMON is the first "AI astronaut" launched by IBM. It helped man land on the moon for the first time 40 years ago.

　　In the "to be no.1" exhibition area where Shangtang Technology cooperates with the Forbidden City, as long as you make a gesture, you can "cross" to take a photo with Confucius in ancient times, and you can also use face recognition technology to "transform" into a scholar who is in Beijing to take an exam and take an ancient imperial examination.

　　The artwork "Nanjing West Road Style Migration" created by young painter Li Yuansu using AI technology is also a popular interactive project in the exhibition hall. When the audience enters the exhibition area, the camera will capture the corresponding pictures, which are processed by human detection, background segmentation, human-computer interaction and other technologies, and then the computer will combine the study of the painter’s past works and make real-time creation. On the display screen, the audience can see their own figure and become a stylized image in real time. Although these works are only based on the creation of man-machine cooperation, it is undeniable that AI has penetrated into all aspects of human life.

　　2 Interaction triggers reverse trigger: artists begin to think about "how to express intelligent objects"

　　In fact, in the AI era, the integration of human artistic creativity and AI has become a "new trend", and there have been many interactions between them.

　　In 2017, Google launched Autodraw, a tool that can help people draw stick figures. With just a few strokes, the algorithm can identify what you want to draw and optimize it for you. In 2018, the digital creation studio OUCHHH launched an exhibition named "Poetic AI" at the Paris Art Center, France. By learning the articles of scientists who have changed human history, AI transcoded them into words and images, and then projected them into a space of 3,300 square meters through 136 projectors. The effects of light and motion are also automatically generated through preset algorithms, which can provide an immersive experience combining sound, light and shadow for the audience. Also in 2018, at Christie’s auction, the painting "The Portrait of Edmund Bellamy" created by AI was taken at a price of 432,500 US dollars (about 3.01 million yuan).

　　Interestingly, Li Yuansu pointed out a new phenomenon: artists used to like to express figures, flowers, birds and mountains. When AI appeared, some people began to think about "how to express intelligent objects". "In the past, artists may describe the sparkling water, but what excites contemporary artists may be the flashing LED lights and the computing interface of supercomputers." He believes that "these are the reverse triggers of artificial intelligence on artistic creation inspiration."

　　3 Constantly unlocking new skills: Microsoft Xiao Bing, who can write poetry and sing, also learned to draw.

　　In May 2014, Xiao Bing, an intelligent chat robot launched by Microsoft (Asia) Internet Engineering Institute, was born. This is a girl who pays attention to EQ and takes an emotional route. In the past five years, Microsoft Xiao Bing has been unlocking new skills. She published a collection of poems, learned to sing, can host programs, and learned to draw as an "supernumerary" graduate student of the Central Academy of Fine Arts. At the World Artificial Intelligence Conference, Xiao Bing’s paintings attracted many audiences with unique style.

　　The painter Xiao Bing’s debut time is not long, but he has become a minor celebrity. In May this year, Microsoft Xiao Bing, under the pseudonym of "Xia Yubing", exhibited her works for the first time at the 2019 Graduate Exhibition of Central Academy of Fine Arts, and became an "supernumerary" master graduate of Central America.

　　After graduation, Microsoft Xiao Bing is "non-stop". From June 15th to July 15th this year, the cross-border exhibition of contemporary art "Xiao Bing, Painting for a Period" was held in Liangzhu Culture and Art Center, Hangzhou, featuring paintings created exclusively by Xiao Bing and VR immersive works created interactively by new media artists Zhou Linwei and Xiao Bing. Later, Xiao Bing’s first solo exhibition "Possibility World" was held in the Art Museum of the Central Academy of Fine Arts.

　　Now just go to Xiao Bing’s homepage and enter a simple text on the page to activate her, and you can entrust Xiao Bing to make a painting for you. After extracting intention, stimulating creative inspiration, selecting content theme, trying to compose the picture, drafting line drawing modeling, painting the bottom color, deepening the picture level and polishing the details repeatedly, the painting created by Xiao Bing can be released in three minutes.

　　According to the Microsoft Xiao Bing team, Xiao Bing spent 22 months studying the paintings of 236 famous painters in the history of art. When inspired by texts or other creative sources, Xiao Bing can independently complete 100% original paintings. This originality is not only reflected in composition, but also in color, expressiveness and details. Li Di, the head of Xiao Bing’s team, stressed: "Xiao Bing’s painting ability is not based on filters and migration technology — — That is, search the picture of the corresponding theme from the landscape library, and then use the oil painting filter to process it. " He said that painting model is the third kind of AI model after text and pronunciation, and it is also the longest and hardest project.

　　Qiu Zhijie, Dean of Experimental Art College of Central Academy of Fine Arts, spoke highly of Xiao Bing’s works. "Xiao Bing’s works are diverse in style, skillful in technique, fast in painting and large in quantity, from which we can find many good paintings." Qiu Zhijie said, "We set several aesthetic rules for artificial intelligence, and it will use these rules to combine endless changes. If you set it to ‘ Do not repeat ’ The instructions, AI actually has a greater chance to get rid of the routine. "

　　Xiao Bing is developing towards the route of piano, chess, calligraphy and painting, which makes her look more like a "person". In May 2017, Xiao Bing’s original poetry collection "Sunshine Lost Glass Window" was published, which is the first poetry collection created entirely by AI in human history. This year, Xiao Bing and the poet’s poetry collection "Flowers are Silence of Green Water" came out, which is also the result of writing poems jointly by man and machine. About 200 poems were finally selected from 6,000 submissions.

　　However, whether it is poetry or painting, Xiao Bing’s works still clearly bear traces of AI. For example, there is a lack of effective transition between poetic images, irrationality between logics, and inconspicuous poetry, etc. Although Xiao Bing’s team said that Xiao Bing’s paintings are close to human level, the audience can still see with the naked eye that Xiao Bing’s works have no three-dimensional sense of oil painting, appear more flat, and can’t see the lines of a painting.

　　4 human beings are hard to be replaced: AI singers who lack emotion are hard to impress people.

　　On the evening of August 29th, when the virtual singer Tianyi Luo’s singing sounded in the Shanghai Benz Cultural Center, the familiar Jasmine seems to be a little strange. As the most famous virtual singer in China, Tianyi Luo has held ten thousand people’s holographic concerts for three years in a row, creating a precedent of "domestic virtual singer AR live broadcast on a large variety show".

　　Then came the piano robot Special Olympics from Italy. I saw his 53 fingers dancing on the keys quickly and accurately, playing and singing Perfect by British singer ed sheeran, Special Olympics’ hard-core head and unique singing voice, leaving a different memory for the audience.

　　At the first audio-visual feast of AI and art in China, AI brought us into a brand-new art world. Man-machine interaction, motion capture, virtual reality, holographic images, three-dimensional images and other technical means are combined with music, dance, opera and other artistic forms to jointly create a mirror world of light and shadow, and a visual spectacle of illusion.

　　However, when you listen carefully, the songs of Tianyi Luo and Special Olympics are still a little less natural and agile. Although the ups and downs of the vocal range can be realized by AI, the free flow and emotional expression of human voices can never appear on AI robots.

　　Artificial intelligence has greatly broadened the boundaries of art and made machines more like people. But "0" and "1" can’t teach machines to have unique human emotions. Without the artistic expression of emotion, it is difficult to really hit the hearts of the audience, and of course, it also makes people realize the "irreplaceable" of human voice.

　　On the Zhihu, someone asked: "What is the significance of developing AI production art?" Xiao Yao, the author of the science fiction "AI Trek", replied: "Artists will not disappear. AI will only eliminate some guys who are not innovative, and those who are truly capable and talented will only get better and better. If the artist’s creative enthusiasm will be hit by AI, it only means that it is normal for these guys to be eliminated. It’s like the appearance of a car is bound to hit ‘ Camel Xiangzi ’ Children, and part of ‘ Camel Xiangzi ’ I learned a driver’s license and became a taxi driver. "

　　interlinkage

　　Dialogue with Li Yuansu — —

　　Art still needs to return to the level of humanity and flesh.

　　Yangcheng Evening News: When did you start using AI for artistic creation? Are you satisfied with the results of this creation?

　　Li Yuansu: This is the first time I have combined with AI to create. With the theme of artificial intelligence, each of the three works has a clear experimental purpose: the mural of "AI Hegemony" is a struggle between hand painting and digital printing on the old media of "mural"; The game "AI Hegemony" uses the same material as murals, and the form is the game interaction mode in the "pre-AI era"; "Nanjing West Road Style Migration" uses contemporary cutting-edge AI technology. In this work, AI learns the artistic style independently and makes his own real-time creation. The three works show the aesthetic characteristics of different media from manual to digital, from static to interactive. The final result is to meet the expectations.

　　Yangcheng Evening News: What new possibilities does AI offer for artistic creation?

　　Li Yuansu: If we sum up the traditional appreciation of art as a group of relationships between artists and audiences, then this group of relationships in the AI era will be expanded, and we will have various combinations such as "machine creation-audience watching", "artist creation-machine watching" and "machine creation-machine watching". Of course, it is not easy to discuss whether machines can produce emotions and how to define emotions, but it is my purpose to do these works to explore along these lines.

　　Yangcheng Evening News: What do you think of the future artistic creation trend?

　　Li Yuansu: Now it seems that the role of artificial intelligence technology in human art will not be subversive. The role of AI in human art is expansion. It brings unprecedented media relations, creative materials, inspiration and great help. It allows human beings to better understand themselves in the mirror of science and technology. However, for the "machine art", recent years may only be the time when it originated.

　　Yangcheng Evening News: When artistic installations can continuously create works without the participation of artists, how should we treat these works of art? Is the painting made by artificial intelligence a real work of art?

　　Li Yuansu: The relationship between art and audience is consensual, and art is a condensed form of emotional experience. So, as long as someone is willing to admit that it is a work of art, it is. This is in essence. However, whether it is recognized academically or accepted by the art market will definitely lead to different situations and twists and turns.

　　Yangcheng Evening News: Is the essence of science and technology art art or technology? Will future artists be replaced by creative artificial intelligence machines?

　　Li Yuansu: Science and technology art is still art. In fact, today’s art is also inseparable from "technology". For example, canvas is a textile technology, and pigments need mineral and chemical technology, including photography and video, which were once high-tech — — Humans will always take everything at hand to create art and "play" art.

　　Artistic creation will not be replaced by machines. For creators, the first thing in art is to serve themselves, and it is an act of expressing emotions and recording experiences. In the contemporary era when information technology subverts the way of life, we are still trapped in the warmth of manual work. At present, only by returning to the level of humanity and flesh can the new media art gain the recognition of more audiences, and the pure aesthetics of science and technology is cold to most people.