The difficulty level is hugely influenced by the school you attend. Another factor is how much you enjoy the material.
Assuming you are smart enough to be successful in either course of study, instead of choosing the field to study, consider what you want to do after you graduate. If you want to be a serious player in either field, you have to go beyond the B.S. For accounting, you need that fifth year so you can sit for the C.P.A. exam. That's a tough one; most people have to retake at least part of it before they pass. For finance, an MBA would serve you well.
Do you want to work for a consulting or accounting firm? Or would you rather work for a large company that is the consumer of the services from these firms. In a consulting or accounting firm, you will learn a lot in the first few years on the job. You will also work very hard, many hours. If you are successful, though, many career opportunities will be available to you.
Finally, what are your ambitions. Do you want to be a senior executive, work 60 hours a week, get fired a few times in your life, and make lots of money? Or would you rather a steady, dependable job where you get your weekends off, don't have to think about work when you are off, but still feel a sense of success and professional satisfaction in your work? (No wrong answer here, but it is good to be honest with yourself about this.)
Oh, one last thought. You don't have to make a decision right now. If your grades are good, and you have the option to attend a competitive school, start in finance, if you can. Then, after you have taken a couple of courses in each area, it will be easy to switch to accounting if that is what you want.
Good luck!